Unity Inference Engine

• Understand how to configure and run AI models directly on-device using the Unity Inference Engine.
• Learn how to convert, serialize, and quantize .onnx models into optimized .sentis assets for faster performance.
• Implement model warm-up routines to prevent stutters and ensure smooth runtime initialization.
• Optimize inference with GPU-based Non-Max Suppression (NMS) and Split Over Frames scheduling.
• How to transform 2D object detections into spatially anchored 3D visualizations using DepthTextureAccess.
• Run on-device SLMs with Unity Inference Engine.

using Unity.InferenceEngine;
using UnityEngine;

public class ModelConverter : MonoBehaviour
{
    public void QuantizeAndSave(Model model, string path)
    {
        ModelQuantizer.QuantizeWeights(QuantizationType.Float16, ref model);
        ModelWriter.Save(path, model);
    }
}

1. Import your .onnx model.
2. Choose your Quantization Type (None, Float16, Uint8).
3. Enter the desired path and name of your converted model
4. Press Convert to Sentis.

1. In the Project window, select your ONNX model.
2. In the Inspector, click Serialize to StreamingAssets.
3. Unity generates a .sentis file inside your StreamingAssets folder.

using Unity.InferenceEngine;
Model model = ModelLoader.Load(Application.streamingAssetsPath + "/mymodel.sentis");

• Faster load times and smaller project size
• Unity-validated format (guaranteed compatibility)
• Easier to share between projects

using Unity.InferenceEngine;
using UnityEngine;
using System.Collections;

public class ModelWarmup : MonoBehaviour
{
    [SerializeField] private ModelAsset modelAsset;
    [SerializeField] private BackendType backendType = BackendType.CPU;

    private void Start()
    {
        StartCoroutine(WarmUpModel());
    }

    private IEnumerator WarmUpModel()
    {
        var model = ModelLoader.Load(modelAsset);
        var worker = new Worker(model, backendType);
        var input = new Tensor<float>(new TensorShape(1, 3, 224, 224));
        worker.Schedule(input);
        yield return null;
        worker.Dispose();
        input.Dispose();
        Debug.Log("Model warmed up and ready");
    }
}

• Warm up before gameplay starts.
• Keep the worker alive across frames.
• Dispose workers only on scene unload.
• For multiple models (for example, STT + detection), warm them up sequentially.

• GpuNMS.cs
• NMSCompute.compute

• SmolLM (135M, 360M parameters)
• Qwen 0.5B
• Phi-2, Phi-3

1. Convert your SLM to .sentis format.
2. Create a UnityInferenceEngineProvider asset and set Mode to Chat.
3. Configure OnDeviceLlmConfig with:
   • Vocab file (JSON): Tokenizer vocabulary file
   • Merges file (TXT): BPE merges file for tokenization
   • Tokenizer config file (JSON): Special tokens and tokenizer settings
   • Chat template format (string): Template for formatting prompts; use {0} for the user message and {1} for the system message
   • Max Tokens (int): Maximum tokens to generate (default: 100)
   • Max Prompt Length (int): Maximum prompt length in tokens (default: 512)
   • Architecture fields: Match maxLayers, numKeyValueHeads, headDim, and eosTokenId to the selected model.

• Steps Per Frame: Control tokens generated per frame (50-300 range)
• Max Tokens: Limit response length (default: 100)
• Max Prompt Length: Prevent memory spikes (default: 512)

using System.Threading;
using System.Threading.Tasks;
using Meta.XR.BuildingBlocks.AIBlocks;
using UnityEngine;

public class ChatExample : MonoBehaviour
{
    [SerializeField] private UnityInferenceEngineProvider provider;

    async Task TestChat(CancellationToken cancellationToken = default)
    {
        try
        {
            var req = new ChatRequest("Hello, how are you?");
            var response = await provider.ChatAsync(req, ct: cancellationToken);
            Debug.Log(response.text);
        }
        catch (System.Exception ex)
        {
            Debug.LogError($"Chat failed: {ex.Message}");
        }
    }
}