Oct 24, 2022

Test Metadata Updates and Minimize Risk with Release Holdouts

This feature has been deprecated

Building a high-quality VR app isn’t easy—and once your app is published, it’s important to be aware of the tools that can help your app make a splash on the Meta Quest Store. One of the best ways to stand out from the crowd is by tailoring your creative assets and metadata to entice your target market—but iterating assets can be risky business without the right data or flexibility.

We’ve recognized the need to turn accessible insights into asset performance, and our team has worked on effective and flexible solutions that can help you build an asset strategy that drives conversions. Since launching in June, our A/B Testing Tool has helped developers drive double-digit conversion increases by comparing the performance of assets shown on their apps’ PDPs.

Now, we’re thrilled to announce that we’ve gone a step further with our new metadata release holdout feature. As of October 10, 2022, new metadata submissions for apps listed in the Meta Quest Store will have the option to automatically use a release holdout. This feature lets you test your new metadata with a portion of audiences and gain detailed insights. Below we dive into how the release holdout works and how you can start using it today.

How Release Holdouts Can Boost Your Strategy

When making metadata updates, you want to feel certain that your changes will lead to improvements in PDP visits and conversions. Using insights from previous A/B tests may shed light on what types of metadata resonate with people, but ultimately it can be difficult to know for certain how your changes will impact sales.

Release holdouts can offer you peace of mind knowing that all changes to critical PDP elements like Cover Art, Descriptions, or Logos will be tested to ensure they don’t cause a major negative impact to your app sales. This means that, by default, the changes you make will only impact a subset of users before reaching 100% of your user base, thereby protecting your app’s Store performance. Release holdouts provide a solution to potentially missed conversions and minimize risk by using our A/B testing framework (read more about A/B tests here).

We’ve added a Release Holdout checkbox to the metadata update flow, which is selected by default for all developers. If the checkbox is selected and your submission is approved, 50% of your users will continue to see your previous metadata for up to 14 days in order to measure the change to your Store conversion rate. If you’re sure that you want to release your metadata updates to 100% of audiences, you can also uncheck the Release Holdout checkbox during the metadata submission flow.

While our A/B Testing Tool provides testing functionality for specific assets within your PDP, the release holdout feature supports testing your PDP as a whole. Release holdouts allow you to test many types of metadata on your app’s PDP, but they don’t support fields like Pricing, App Name, Specs, and Details. If you want to update these fields, first unselect the Release Holdout checkbox.

Get Started with Release Holdouts

When you create a new app submission in the Developer Dashboard⁠ under Distribution → App Submissions, you’ll see a Release Holdout checkbox and an informational popup stating, “Your submission release will include a holdout period.”

There are three potential outcomes of the release holdout:

If the change has a positive impact on conversion rate, or the negative impact is less than a 3% drop in conversion, the update will be automatically published for all users.

If the change leads to a drop in conversion rate by 3% or more, all users will automatically revert to the previous metadata, and you can choose if you want to publish the update for all users.

If your A/B test doesn’t reach statistical significance within 14 days after your test starts, your metadata update will be automatically published for all users. For more information on statistical significance, check out our A/B Testing FAQs.

To help you stay informed, you will be notified via email for all of the outcomes listed above. For a deeper look into your updated metadata’s impact on conversion rate, you can also view results in the A/B Testing Tool under the “Holdout Releases” tab.

We’re excited to offer you new features that can help you optimize metadata performance in the Store and provide solutions that minimize risk—allowing you to iterate and experiment with your assets more freely. For more information, see our A/B Testing Tool documentation. If you have feedback, let us know in the Developer Forum⁠.

Test Metadata Updates and Minimize Risk with Release Holdouts

This feature has been deprecated

How Release Holdouts Can Boost Your Strategy

Get Started with Release Holdouts

Get the latest updates from Meta Horizon.