Developer Perspective: Improving Memory Usage and Load Times in UE4

This post has some notes from our work at Turtle Rock on improving loading times and memory usage for Journey of the Gods and Face Your Fears 2 for the Oculus Quest. These changes improved our overall performance and helped us quickly get a loading spinner on screen when the games were launched. The first section has a few quick things you can do to change what and when your game loads. We then go over the asset reference system and a number of tools you can use to get a clearer picture of your game’s memory usage.

Review your project’s plug-ins and disable anything that the game doesn't need. These all take up initialization time early in the application life cycle, and can consume memory and other resources. When you launch the game, you can see the list of plugins that are getting initialized early in your log. Some will add permissions to your manifest, so removing those is a win-win-win. Get aggressive here! If you disable something you really need it is usually caught at build time or will be immediately obvious in QA.

Configure your project’s “Game Default Map” to load an empty map which uses the default UE4 pawn, game mode, etc. classes. These will have the fewest asset dependencies so it will load quickly, enabling you to display the loading spinner and transition to a main menu map. This is when we do our entitlement verification check, and if it fails, we redirect the player to an empty map with a modal error message box that explains the failure and exits the application.

Depending on your application, one way to get the loading spinner up in less than 4 seconds is to have certain plugins load later by changing the "LoadingPhase" in their .uplugin file. For example, we changed the Wwise plugin to load during "PostEngineInit" after the loading spinner was displayed.

If your project is configured to package data in to OBB files, UE4 has some OBB verification that can add 5-10 seconds of data checks to your application launch time. You can disable this with a config change:

 Project/Config/DefaultEngine.ini
[/Script/AndroidRuntimeSettings.AndroidRuntimeSettings]
bDisableVerifyOBBOnStartUp=True

Avoid using the ConstructorHelpers::FObjectFinder and ConstructorHelpers::FClassFinder in your C++ class constructors. These cause assets to be loaded when the CDO classes are initialized, which happens very early in your application launch sequence. If the constructor static loads a data table and the data table has references to all of your characters, you will load all of your character data before you ever see a loading spinner! Instead of ConstructorHelpers, use FSoftObjectReferences (or whatever they're called in your version of UE4), and use their async loading system to load those assets when it makes sense for your application.

You can decrease your memory usage by lowering the size of the texture streaming pool. It is likely that many of your assets will just have a diffuse texture, without any normal maps, masks for roughness, metallic response, etc. You will probably end up using smaller textures or atlasing your textures. Given this, it’s likely that you can cut the texture streaming pool size in half without affecting the quality of the experience in-game:

Project/Config/Android/AndroidEngine.ini
[SystemSettings]
r.Streaming.PoolSize=500

Reducing shader permutations can help lower your memory usage and speed up launch times. There are individual settings you can toggle based on the features that you need, and Epic has some documentation on the individual settings. In newer versions of the engine you can also discard unused shader quality levels:

Epic has documentation on their Reference Viewer tool that helps show the relationship between your assets. Open your main menu map and check out some of the placed assets in the Reference Viewer tool to get a feel for how the tool works. Remember that your pawn and game mode classes get spawned when this map loads, so it would be good to check those and see what assets they’re bringing along with them. You can help control your asset reference graph by having your Blueprint classes hold references to C++ classes rather than Blueprints (which often have more asset and class dependencies). We have learned from experience that cleaning this up late in a project is very, very difficult. It is much better to do periodic checks of the reference graph and fix problems as they appear.

adb shell "am broadcast -a android.intent.action.RUN -e cmd 'memreport -full'"
adb pull sdcard/UE4Game/YourProject/YourProject/Saved

See our earlier article on UE4 logging and console commands for mobile VR devices.

First, review the section that starts with "Obj List: -alphasort". This will show you the names of Blueprint objects that are currently loaded. Note that the size is provided only for the class data, not their textures, models etc. You want to look for objects that are being loaded on your Main Menu when they aren't needed until later in the game. These loaded objects are increasing your initial load times unnecessarily. Once you've identified objects that shouldn't be loaded, use the Reference Viewer tool to see what is causing them to do so. Remove any unnecessary references to break those dependencies so that they won’t be loaded.

Second, find a section in the memreport file that starts with "Listing all textures". Textures can get large and take up a lot of memory, be on the lookout for uncompressed textures that don't have mips. If a texture doesn't have mipmaps, it will have "NO" in the "Streaming" column:

2048x2048 (16384 KB, ?), 2048x2048 (16384 KB), PF_B8G8R8A8, TEXTUREGROUP_World, /Game/Episodes/S1E1/Items/SmartPhone/SPSIncomingCall_T.SPSIncomingCall_T, NO, 0
//

This means the texture streaming system has to load the entire file into memory, it can't stream in/out individual mip levels as needed. This increases loading times and memory usage. Being marked as PF_B8G8R8A8 also means that it is uncompressed, again this increases loading times and memory usage. There are rare circumstances where this is required (certain fancy VFX may need this), but most will be set this way accidentally.

The next thing you'll want to do is look in the "Memory Stats" section, put on your detective hat, and look for any numbers that are much larger than you are expecting. For example, I saw this during a memory investigation when we were shipping Face Your Fears 2:

125108160  -  Render target memory Cube - STAT_RenderTargetMemoryCube - STATGROUP_RHI - STATCAT_Advanced

Why would we be allocating 125mb for a cube map render target on the Quest? We aren't doing cubemap captures during gameplay, we do that at edit time. Near the bottom of the file was this entry:

0x0 (0 KB, ?), 0x0 (0 KB), PF_Unknown, TEXTUREGROUP_RenderTarget, /Game/Common/Environments/Sky/HouseInt_WindowsB_T.HouseInt_WindowsB_T, NO, 0

A texture that takes 0kb, and it's in the RenderTarget group? That was suspicious. When I opened the asset in the material editor, I saw that it was indeed a 120+mb render target texture. It was being referenced by a material that had been used when experimenting with a feature on the map but was no longer needed. We deleted the assets and immediately got back 120mb of GPU memory!

The UE4 editor has a powerful “Statistics Viewer” that shows you stats about the textures and meshes in your scene:

Here is a sample asset statistics view from the UE4 “Shooter Game” sample application:

You can use this to quickly see if you have textures with the wrong compression / mipmap settings, or are unnecessarily large. It can also provide the triangle count of the meshes in your map. If you haven’t surveyed your assets yet, you can usually find a few interesting surprises with this tool.

Recent versions of UE4 have added a feature called the PSO Cache which helps reduce gameplay hitches caused by runtime shader compilation. This system can be used to pre-compile the shaders so they’re ready when objects are rendered, and help you avoid those nasty hitches. The only downside is that it can take a long time to precompile those shaders - 30 seconds or more is not uncommon. We decided to control when this happens, so that we can show users an “Optimizing Assets for your System” type of modal UI.

To accomplish this, you can add some logic to defer the pipeline cache compilation. First call FShaderPipelineCache::PauseBatching() inside your UGameinstance::Init() (or earlier if needed). Then load into an empty map and bring up a loading screen to let the player know you're doing a onetime asset optimization. Once the loading screen is up, call: FShaderPipelineCache::SetBatchMode(FShaderPipelineCache::BatchMode::Fast) and FShaderPipelineCache::ResumeBatching(). Bind a function to the FShaderPipelineCache::OnPrecompilationComplete delegate to get notified of when the shaders are compiled, then close the loading screen and transition to your main menu map.

It is important to note that in UE 4.21 we encountered an issue which would cause it to precompile the shaders every time the application was launched. It should only need to compile them once, then it should re-use the compiled shaders (which is much faster!) on subsequent launches of the application. If you encounter a similar bug, feel free to discuss it with Epic on UDN.

At this point, your application should have a lower install footprint, better launch times and improved memory usage. If you’re still having problems getting a loading spinner on screen quickly, you should look at what your game is logging out when it starts. The logs will help you track down what work is being done before the spinner is shown and dig into any remaining systems that are causing you problems.

If you want to discuss anything we’ve covered in these blogs or just talk about VR in general, jump in to our forums, join us on Discord or hit us up on Twitter @TurtleRock! You can also email me directly - brock [at] turtlerockstudios [dot] com

Brock Heinz
Lead Programmer
Turtle Rock Studios