Creative A/B Testing FAQ
Updated: Dec 16, 2024
The creative A/B testing tool lets you compare your existing assets on your app’s product detail page (PDP) with test assets to determine a winning asset strategy that improves conversions. Asset performance is measured competitively to identify the winning assets that generate statistically significant improvements. If your test assets are declared winners, you can publish them to your app’s PDP.
Who can run creative A/B tests?
Developers with apps published in the Meta Horizon Store can run A/B tests using the creative A/B testing tool in the
Growth >
Creative A/B Testing tab of the
Developer Dashboard.
What variables can I test?
Creative A/B testing allows you to compare five PDP variables: video trailer, description (short and long), cover art (square and landscape), screenshots, and immersive layers.
- Video trailer:
- The video that displays at the top of your app’s PDP.
- The thumbnail image that previews your video trailer.
- Description
- Short Description: Description that is shown throughout the platform, primarily in VR.
- Long Description: Description that is shown throughout the platform, including the Store.
- Cover art
- Cover art (square): 1:1 static image that’s displayed across platforms to market your app to users in the Store.
- Cover art (landscape): 16:9 static image that’s displayed across platforms to market your app to users in the Store.
- Screenshots
- Each image should represent a unique scene, highlighting the best part of the experience.
- Note: Reordering existing screenshots does not require review and will be automatically approved for testing.
- Immersive layers
- Immersive layer (backdrop): A large image that sits behind the panel app.
- Immersive layer (logo): A small image displayed above the panel app that features the logo of your app.
- Immersive layer (left/right object): a significant object or character from your app that sits to the left or right of the panel app. Two object images are supported per app, and you have the option of splitting them between the left and right side.
What do creative A/B tests measure?
Creative A/B tests compare the performance of PDP assets based on conversion rate, reach, unique clicks, and unique conversions. If a statistically significant performance difference is found between your assets, the better performing asset(s) will be identified as the winner.
How are my test results deemed statistically significant?
Statistical significance is the confidence that a relationship exists between user behavior and the assets being tested, and that the relationship can be attributed to variables other than chance. It is achieved once enough visitors have viewed your assets, and the conversion rate between the assets maintains an error rate of less than 5% or a 95% confidence interval. The amount of visitors and error rate may vary, but these metrics help ensure that your results are significant and can be used to verify your hypothesis.
How long do creative A/B tests take?
All creative A/B tests run for a maximum of 30 days. Statistical significance can be achieved in seven days at the earliest. Throughout the test you may view your results in the creative A/B testing tool, or cancel your test at any time. Please note that only one A/B test may run at a time per app and you can’t publish other app metadata changes while a test is active.
How do I manage my tests and view results?
Tests that have been submitted, are running, or completed are managed within the creative A/B Testing tool in the
Growth >
Creative A/B Testing tab of the
Developer Dashboard. It displays general information, including the status and winning asset of each A/B test. You can click the overflow menu on the right side of each item to cancel, or view the results of your test. You will receive email notifications alerting you when your test is approved, requires changes, achieves statistically significant results, or is completed.
Will my test asset automatically publish to my app’s PDP if it’s the winner?
If the Auto Publish option is enabled for your creative A/B test, your test assets will be automatically published to your app’s PDP after your creative A/B test has run for at least 20 days and produces statistically significant positive results (>0% change in conversion rate).
Otherwise, you can manually publish your test assets to your app’s PDP after your test has completed by navigating to the Results page and selecting Publish Test Variant in the bottom-right corner.
Why am I unable to create a new test?
If your app is listed on the Store and the create button is disabled, make sure that you don’t have an active creative A/B test. Only one test can run at a time. In addition, creative A/B tests can not be created if your app has an open metadata submission that is pending or under review.
Can I test multiple PDP variables simultaneously?
Yes, but note that when you’re testing more than one variable (video trailer, description, cover art, screenshots, and immersive layers) you won’t know exactly which asset led to the winning result.
Consider your goals when testing. For example, it may be appropriate to change multiple variables if you are running a campaign in which multiple assets will need to differ. However, if you are trying to understand how a single variable like how a trailer performs, you may want to only test the video trailer.
Can I run multiple creative A/B tests simultaneously?
Creative A/B tests only allow for one test to run at one at a time in order to eliminate overlap amongst user groups.
What proportion of my audience will see my test asset?
Creative A/B tests allow you to choose between testing your assets with 10%, 25%, or 50% of your audience. While you can test with a smaller audience, a 50/50 split helps ensure a higher probability of statistically significant results, and can help you achieve significant results faster.
How long does the submission approval process take?
After submitting your asset, you will receive an email with your approval status in an estimated five to eight business days. Approval times can vary depending on the number of submissions received by the Review team and may take longer. If changes are requested during this time you will also receive an email notifying you to make changes to your test and resubmit.
Some PDP variables bypass the review process:
- Screenshots - reordering existing screenshots will be automatically approved for testing
Does creative A/B testing allow for localized descriptions of my test asset?
While you are not required to localize your assets, you have the ability to create one or more localized descriptions for your test asset if you are testing either short or long descriptions.