Develop

Adding New Providers

Updated: Jul 8, 2026

Learning Objectives

Understand how Providers serve as modular inference backends for AI Building Blocks.

Learn to create custom Providers that integrate seamlessly with existing Agents.

Design intuitive Editor interfaces for Provider configuration and validation.

Implement custom network transports (for example, WebSocket or gRPC) for real-time or low-latency inference.

Extend or adapt Agents to add contextual or multimodal behavior with minimal code changes.

The AI Building Blocks framework is provider-agnostic and fully extensible. You can integrate new inference backends, custom transports, or unique model types simply by implementing standard task interfaces. This design allows your Unity Agents (LLM, Object Detection, STT, TTS) to remain unchanged while switching to any compatible model or service.

Before You Begin

Review existing Providers (Llama API, OpenAI, HuggingFace) before creating your own. Use them as templates. All Providers follow the same task interface patterns and are discovered automatically at install time.

Architecture Recap

Each AI Building Block consists of two layers:

Layer	Role
Agent (MonoBehaviour)	Handles Unity-side orchestration: captures input, triggers inference, and dispatches results.
Provider (ScriptableObject)	Defines the backend — how and where inference happens (Cloud, Local, On-Device). Defines input/output structure.

By creating a new Provider, you can integrate:

A custom REST API or WebSocket inference service.

A new model type (for example, segmentation, diffusion, emotion recognition).

Core Interfaces

Every Provider implements one or more of the following interfaces:

Interface	Purpose
`IChatTask`	Defines text or multimodal LLM chat tasks.
`IObjectDetectionTask`	Handles object detection (boxes, labels, confidence).
`ISpeechToTextTask`	Converts audio input to transcribed text.
`ITextToSpeechTask`	Converts text into playable speech audio.

All interfaces expose a single async method (for example, ChatAsync(), DetectAsync()) that accepts structured input and returns a strongly typed result.

Example — IChatTask Provider

Custom chat provider implementing IChatTask

This skeleton shows the current IChatTask contract for a cloud chat Provider. Use the built-in Llama API, OpenAI, HuggingFace, or Ollama Providers as full reference implementations for provider-specific request and response formats.

using System;
using System.Threading;
using System.Threading.Tasks;
using UnityEngine;

[CreateAssetMenu(menuName = "Meta/AI/Provider Assets/Cloud/My Chat Provider")]
public sealed class MyChatProvider : AIProviderBase, IChatTask
{
    [SerializeField] private string apiKey;
    [SerializeField] private string endpointUrl = "https://api.example.com/v1/chat";
    [SerializeField] private string model = "my-chat-model";
    [SerializeField] private bool supportsVision;

    protected override InferenceType DefaultSupportedTypes => InferenceType.Cloud;
    public bool SupportsVision => supportsVision;

    public async Task<ChatResponse> ChatAsync(
        ChatRequest req,
        IProgress<ChatDelta> stream = null,
        CancellationToken ct = default)
    {
        ValidateConfiguration(apiKey, endpointUrl, model);
        var body = new ChatBody { model = model, prompt = req?.text ?? string.Empty };
        var json = JsonUtility.ToJson(body);
        var raw = await new HttpTransport(apiKey).PostJsonAsync(endpointUrl, json, ct: ct);
        var parsed = JsonUtility.FromJson<ChatProviderResponse>(raw);
        return new ChatResponse(parsed?.text ?? string.Empty, raw);
    }

    [Serializable]
    private class ChatBody
    {
        public string model;
        public string prompt;
    }

    [Serializable]
    private class ChatProviderResponse
    {
        public string text;
    }
}

This provider can be assigned directly to any LlmAgent. For multimodal providers, set supportsVision to true, call PrepareRequestImagesAsync(), and include the prepared image payloads in your provider-specific request body.

Using `AIProviderBase`

All Providers in the framework — including LlamaApiProvider — inherit from AIProviderBase, which provides:

Responsibility	Description
Capability Flags	Exposes `SupportedInferenceTypes` so editor tooling can filter compatible Providers.
Configuration Checks	Provides `ValidateConfiguration()` for required API key, endpoint, and model fields.
Request Helpers	Provides helpers for endpoint normalization, image preparation, and texture encoding.
Parsing Utilities	Provides shared detection parsing helpers used by object-detection Providers.

This shared base class ensures that all Providers behave consistently and integrate seamlessly with Unity’s async execution model.

Registering Your Provider

Your Provider is automatically detected by the AIBlocksInstallationRoutine. To make it appear in the setup wizard:

Add a CreateAssetMenu attribute: Meta/AI/Provider Assets/<YourCategory>

Implement one of the known interfaces (IChatTask, IObjectDetectionTask, and so on).

Save the Provider as an asset in your project.

During installation, the system scans all available Provider assets, groups them by inference type, and displays them in the configuration dialog.

Integrating Custom Transports

The built-in HttpTransport handles standard HTTP workflows, but you can extend it for real-time streaming or gRPC/WebSocket backends.

Example Workflow

Create MyWebSocketTransport.cs.

Implement connection setup, message sending, and receive callbacks.

Use it internally within your Provider instead of HttpTransport.

This lets you build Providers that support live streaming (for example, token-by-token LLM output) or extremely low-latency local inference — without modifying any Agent code.

Extending Existing Agents

Sometimes it’s easier to extend an existing Agent rather than a Provider. You can create wrapper components for custom prompts, pre/post-processing, or contextual awareness. This keeps your Provider unchanged while customizing runtime behavior.

using System.Threading.Tasks;
using Meta.XR.BuildingBlocks.AIBlocks;
using UnityEngine;

public class ContextualLlmHelper : MonoBehaviour
{
    [SerializeField] private LlmAgent llmAgent;
    public string Context = "You are a helpful assistant.";

    public async Task SendContextualPrompt(string userInput)
    {
        var fullPrompt = Context + "\n" + userInput;
        await llmAgent.SendPromptAsync(fullPrompt);
    }
}

→ Next: Troubleshooting and FAQ