Develop

AI Building Blocks - Overview

Updated: Jul 8, 2026

AI Building Blocks bring plug-and-play machine-learning capabilities directly into Unity-based XR projects for Meta Quest. Each block combines a Unity Agent (runtime logic) with a configurable Provider (defining where inference runs: A. in the cloud, B. on a local machine, or C. on the Meta Quest device).

Core AI Building Blocks

AI Building Blocks offer modular functionality across four key categories:

Object Detection

Detect and label real-world objects in passthrough or camera textures using on-device or cloud models.

View Documentation

Large Language Models

Integrate contextual or multimodal AI using Llama, or custom models through any Provider.

View Documentation

Speech to Text (STT)

Transcribe microphone or audio-clip input in real time using state-of-the-art cloud models.

View Documentation

Text to Speech (TTS)

Generate natural-sounding voice output using state-of-the-art ElevenLabs or OpenAI models.

View Documentation

System Recommendations

Unity 6 or newer

Meta Quest 3 or 3s

Meta XR Core SDK v83+ and Meta XR MR UtilityKit v83+ (for Passthrough Camera support)

Stable internet connection when running cloud inference or local inference through HTTP ports

Typical Use Cases

Always check provider and model availability

We do our best to provide you with state-of-the-art providers and up-to-date models, but especially for cloud providers, models may not always be available on the provider's servers. Therefore, always make sure to check the provider and model availability before using them in your experience.

Category	Example	Recommended Provider
Vision	Real-time object detection	Unity Inference Engine / HuggingFace
Language	Language and Vision Requests to LLMs/VLMs	Llama API / OpenAI / Ollama / HuggingFace
Speech	Voice commands or narration (TTS / STT)	OpenAI / ElevenLabs

Architecture Overview

Each AI Building Block consists of two core layers:

Layer	Role	Examples
Agent	Unity runtime component managing input/output and inference calls.	`ObjectDetectionAgent`, `LlmAgent`, `SpeechToTextAgent`, `TextToSpeechAgent`
Provider	`ScriptableObject` defining the inference backend and input/output structure.	`OpenAIProvider`, `HuggingFaceProvider`, `OllamaProvider`, `UnityInferenceEngineProvider`

Example

Prototype using Llama 4 Maverick (Llama API), then switch to Llama 3.3 running on Ollama, or an on-device model, without changing your experience’s logic.

Release Notes

Version 85 and later

New Features:

On-Device LLM Support: Run text generation models (SmolLM, Qwen, Phi) directly on Quest with blocking/non-blocking execution modes

Credential Management: Store API keys centrally via CredentialStorage and validate the connection of your API key and endpoint quickly and reliably.

Enhanced Features:

UnityInferenceEngineProvider: Supports object detection, image segmentation, and on-device Small Language Models (SLM) chat

LlmAgent: Added multimodal image capture events, vision support detection, and auto-warmup for on-device providers

ObjectDetectionAgent: Configurable detection frequency (detectEveryNFrames) for performance tuning (0 = never/only when manually triggered, 1 = every frame, 2 = every other frame, and so on)

IChatTask: Expanded with ImageInput, ChatRequest, ChatDelta, and ChatResponse types for multimodal and streaming support

Breaking API changes when migrating from v83:

IChatTask.ChatAsync(): Method signature changed from RunChatAsync(string prompt, CancellationToken token) returning Task<string> to ChatAsync(ChatRequest req, IProgress<ChatDelta> stream = null, CancellationToken ct = default) returning Task<ChatResponse>

Migration: Replace var text = await provider.RunChatAsync(prompt, token) with var response = await provider.ChatAsync(new ChatRequest(prompt), ct: token) and access text via response.text

Task Interface Methods: Standardized method naming from RunTaskAsync() / RunChatAsync() to task-specific methods like ChatAsync() and DetectAsync()

Version 83 (Initial Release)

Initial release with four core Building Blocks (Object Detection, LLM, STT, TTS), provider system (cloud, local, on-device), task interfaces, depth integration, GPU-accelerated NMS, and event-driven architecture.