Meta Developers

Virtual Keyboard provides a glTF render model which enables the application to have control over the rendering of the keyboard. To accurately represent the current state of the keyboard as managed by the runtime, the application should obtain texture and animation updates through the extension API in order to apply them to the model.

Importing glTF Models

To integrate Virtual Keyboard, the application will require a glTF loader that is compatible with the 2.0 specification.

Concretely, the model returned by the extension will be a memory buffer in binary glTF (i.e. .glb extension) format. The model may include:

Static meshes
UVs
Normals
Tangents
Vertex colors
Materials
Textures with alpha transparency (embedded)
Animations (for TRS and weights)
Morph targets (for position and UVs)

Custom Image URI Handling

The Virtual Keyboard runtime dynamically generates pixel data for textures used for character glyphs and visual effects. These textures are represented using a custom URI format in the glTF render model.

When parsing the glTF model data, the application must provide a custom URI handler implementation. This handler will be responsible for creating writable textures that can be referenced by a unique texture ID for later use.

The custom image URI will have the following format: metaVirtualKeyboard://texture/{textureID}?w={width}&h={height}&fmt=RGBA32

During each update tick, the application should call xrGetVirtualKeyboardDirtyTexturesMETA to get the list of texture IDs that need to be updated. For each of the IDs, the application should call xrGetVirtualKeyboardTextureDataMETA to retrieve the actual pixel data to apply to the corresponding texture created by the custom URI handler.

For an example on how this is done, please see the VirtualKeyboardModelRenderer class provided in the XrVirtualKeyboard sample project.

Additive Morph Target Animations

The glTF specification allows an animation channel to target a specific property of a node using the path. Valid path names include “translation”, “rotation”, “scale”, and “weights”. However, it’s important to note that by default, the glTF specification does not provide built-in support for individual manipulation of each morph target weight.

To ensure that the dynamically generated textures, containing suggestion words of varying lengths, can be accurately mapped onto the keys of the keyboard model, Virtual Keyboard utilizes additive weight animations in order to control the mesh vertex positions and UVs.

To facilitate this, an integer value named additiveWeightIndex may be written in the “extras” property of an animation channel in the returned glTF model. This value indicates the specific morph target weight to which the animation should be additively applied. If the additiveWeightIndex property is absent or has the value -1, the animation should by default update all the morph target weights instead.

Here is an example with an animation channel containing the additive weight index property:

{
    "animations": [
        {
            "name": "Suggestion Key Label #1",
            "channels": [
                {
                    "sampler": <sampler_id>,
                    "target": {
                        "node": <node_id>,
                        "path": "weights"
                    },
                    "extras": {
                        "additiveWeightIndex": 3
                    }
                },
                ...

            ],
            "samplers": [
                ...
            ]
        }
    ]
}

Each model node representing a key label contains a mesh quad with four vertices. For each of these nodes, there are a total of 16 morph targets. The first 8 morph targets control the XY coordinates of the vertex positions (with the Z coordinate remaining constant), while the remaining 8 morph targets control the UVs.

During each update tick, the application should call xrGetVirtualKeyboardModelAnimationStatesMETA to get the list of animation-index-and-fraction pairs to apply to the model.

For an example on how to handle the additive morph target animations, please see the VirtualKeyboardModelRenderer class provided in the XrVirtualKeyboard sample project.

Best Practices

It is worth noting that parsing the glTF model can be a time-consuming process. Therefore, it is advisable for applications to perform this operation during the initialization phase rather than on demand. By default, the loaded render model is hidden, meaning that the root node has a scale of 0. When text entry is required, you can request the runtime to show the render model. This approach helps optimize the performance and responsiveness of the application.

Secondly, we expect applications to get and apply all texture updates each frame before retrieving the animation states, as our current implementation may queue up new animations during texture data generation that should be applied in the same frame.

Finally, since the geometry of a single node mesh can be updated by multiple animations within the same frame (due to additive morphing), it is recommended to mark the nodes that require a geometry update as dirty rather than regenerating the mesh immediately. Once all the necessary animation updates are complete during each frame update, regenerate the mesh for the marked dirty nodes.

Did you find this page helpful?