Agents and Building Blocks

• Understand how Agents coordinate runtime data flow and events in AI Building Blocks.
• Learn how Providers handle model inference across Cloud, Local, and On-Device backends.
• Connect multiple Agents to build modular, event-driven, and multimodal AI pipelines.
• Combine vision, speech, and language Agents to create interactive XR experiences.
• Extend or customize existing Agents using UnityEvents and wrapper components without modifying Providers.

• ObjectDetectionAgent: Runs inference and returns structured detection data.
• ObjectDetectionVisualizer: Renders bounding boxes or 3D meshes in the scene.

1. The Agent captures a frame from PassthroughCamera.
2. It sends the texture to the assigned Provider (for example, UnityInferenceEngineProvider or HuggingFaceProvider).
3. The Provider returns detection results (bounding boxes, class labels, confidence scores).
4. The Visualizer draws the results in world space.

1. EnvironmentDepthManager: Retrieves per-eye environment depth textures from the Meta Quest system and makes them available globally for other components.
2. DepthTextureAccess: Reads these depth textures from the GPU (via EnvironmentDepthManager), copies them into a CPU-accessible array, and exposes per-frame depth data.
3. ObjectDetectionAgent: Uses the 2D bounding boxes from its model output and maps them to world-space using the latest depth information from DepthTextureAccess.
4. ObjectDetectionVisualizer: Renders 3D bounding boxes or markers in the scene at the world-space positions provided by ObjectDetectionAgent.

// OnBoxesUpdated is a C# Event - use += operator
agent.OnBoxesUpdated += (detections) =>
{
    foreach (var d in detections)
        Debug.Log($"Detected {d.label} at position {d.position}");
};

• LlmAgent: Manages text or multimodal (image and text) conversation flow.
• LlmAgentHelper: Connects LLMs/VLMs with speech agents (STT/TTS) or other custom logic.

1. SendPromptAsync() forwards user input to the Provider (Llama API, OpenAI, Ollama, and so on).
2. The Provider streams text tokens asynchronously.
3. The Agent emits onResponseReceived and onStreamingDelta events.
4. Other Agents (like Text to Speech) can subscribe to stream updates for immediate feedback.

// UnityEvents - use .AddListener() method
speechToTextAgent.onTranscript.AddListener(transcript =>
{
    _ = llmAgent.SendPromptAsync(transcript);
});
llmAgent.onResponseReceived.AddListener(textToSpeechAgent.SpeakText);

• SpeechToTextAgent: Captures microphone or audio clip input.
• Works with Providers implementing ISpeechToTextTask.

1. The Agent records audio or receives an AudioClip.
2. Audio is encoded and sent to the Provider (OpenAI, ElevenLabs).
3. The Provider transcribes it to text and triggers onTranscript.

// onTranscript is a UnityEvent - use .AddListener() method
speechToTextAgent.onTranscript.AddListener((string transcript) =>
{
    Debug.Log($"User said: {transcript}");
});

• Supports real-time or clip-based transcription.
• Some Providers offer streaming recognition for live captions.
• Optional language code field for multilingual setups.

• TextToSpeechAgent: Converts text into audio clips using ITextToSpeechTask Providers.

1. The Agent receives text via SpeakText().
2. It sends the request to the Provider (for example, ElevenLabs, OpenAI TTS).
3. The Provider returns or streams an AudioClip.
4. The Agent plays it automatically or exposes it for custom playback.

// onResponseReceived is a UnityEvent - use .AddListener() method
llmAgent.onResponseReceived.AddListener(textToSpeechAgent.SpeakText);

• Event Types: AI Building Block UnityEvents use .AddListener() for subscription. Some UnityEvents are exposed through PascalCase public properties such as OnDetectionResponseReceived; the OnBoxesUpdated callback is a C# event and uses the += operator.
• MRUK-only events (like OnBoxesUpdated) are wrapped in compile guards and only available if MRUK is installed and the Passthrough Camera is available.
• LlmAgentHelper connects existing events (onPromptSent, onResponseReceived) for simplified wiring but doesn’t introduce new ones.

[Microphone Input] → SpeechToTextAgent → LlmAgent → TextToSpeechAgent → [Audio Output]

[Camera Frame] → ObjectDetectionAgent → LlmAgent (context injection) → TextToSpeechAgent

using System.Collections.Generic;
using Meta.XR.BuildingBlocks.AIBlocks;
using UnityEngine;

public class HighConfidenceDetectionResponder : MonoBehaviour
{
    [SerializeField] private ObjectDetectionAgent objectDetectionAgent;

    private void OnEnable()
    {
        objectDetectionAgent.OnDetectionResponseReceived.AddListener(HandleDetections);
    }

    private void OnDisable()
    {
        objectDetectionAgent.OnDetectionResponseReceived.RemoveListener(HandleDetections);
    }

    private void HandleDetections(List<BoxData> detections)
    {
        foreach (var detection in detections)
        {
            if (detection.label.Contains("person"))
            {
                Debug.Log($"Detected {detection.label} at {detection.position}");
            }
        }
    }
}