Spatial Scanner overview

• Develop enjoyable experiences that help developers think about new ways to utilize the Spatial SDK.
• Create an open-source project that demonstrates Spatial SDK’s Passthrough Camera API and helps developers understand how to implement it.
• Provide documentation that helps developers navigate the project and understand the possibilities of Spatial SDK.

1. Passthrough Camera API: access the forward-facing cameras on the Quest 3 and Quest 3S for livestreaming or CV inference.
2. ML Kit Object Tracking: perform realtime on-device detection and labeling of objects in the user’s field-of-view.
3. Llama 3.2 11B Vision Invocation: utilize Meta AI as a cloud service to get more in-depth details of an object detected in the user’s surroundings.

1. Login to the AWS console, and navigate to the Bedrock service.
2. Near the bottom of the left side navigation column, select the Bedrock configurations/Model access menu item. Scroll down to make sure that access has been granted to the Llama 3.2 11B Vision Instruct model. If not, select the “Modify model access” button at the top of the page, and follow the steps to gain access.
3. Because the AWS foundation model Llama 3.2 Instruct uses geofencing, we’ll also need to enable a Bedrock Cross-region inference. Navigate to the Cross-region inference page in the AWS console - also accessible via the left side navigation column when viewing the Bedrock service page. Search for the model US Meta Llama 3.2 11B Instruct, and select the name of the inference profile result. Make sure that the status is “Active”, or follow the AWS documentation on enabling cross-region inference.

1. Create a custom permissions policy which allows the bedrock:InvokeModelWithResponseStream action on the foundation model and inference profile resources for the Llama 3.2 Vision Instruct model. Follow these instructions to create the policy, and remember the name you’ve given the policy – e.g. InvokeBedrockLlama3Model. Your policy JSON should look similar to this:

    ```
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "VisualEditor0",
                "Effect": "Allow",
                "Action": [
                    "bedrock:InvokeModelWithResponseStream"
                ],
                "Resource": [
                    "arn:aws:bedrock:*::foundation-model/meta.llama3-2-11b-instruct-v1:0",
                    "arn:aws:bedrock:*:*:inference-profile/us.meta.llama3-2-11b-instruct-v1:0"
                ]
            }
        ]
    }
    ```
   
2. Next, follow the instructions here to create and configure an IAM user.
   1. You may assign a descriptive User name of your choice – e.g. LlamaServiceAccount
   2. It is not necessary to grant the user access to the cloud console.
   3. For this new user, assign the policy you just created to the new user.
3. Finally, you must generate an access key for this new AWS IAM user, and save those credentials.
   1. Navigate to the IAM service dashboard, and then select the Access management/Users menu item on the left side navigation bar.
   2. Find and select the user you just created to view their account page.
   3. Select the Security credentials tab on the mid-page navigation, and scroll down to the Access keys card.
   4. Select the “Create access key” button on the right side.
   5. Select the “Other” option as your use-case.
   6. Add a description – e.g. Service account access key for invoking Llama 3.2 in Spatial SDK project.
   7. Select the “Create access key” button, and save the “Secret access key” value someplace secure, as you will not be able to access that value after you leave the webpage. You’ll need the access key and secret access key in the below step on adding your app secrets to the Android project.
4. Note that you should always follow best practices for keeping your access key secure.

1. Duplicate the secrets.properties.example file, and rename it to secrets.properties. In this new file (pictured below), replace the values with your access tokens, and the AWS region of the service account (e.g. us-east-2).

    # AWS Bedrock keys
    AWS_REGION=XX-XXXX-X
    AWS_BEDROCK_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXX
    AWS_BEDROCK_SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
   

• res/drawable/rounded_box_outline.png – used to render a 9-slice outline around detected objects.
• res/layout/ui_camera_view.xml – used to display a debug view of the device camera feed and detected object outlines in front of the user.
• res/layout/ui_camera_status_view.xml – a simple Android view that displays the current camera scanning status.
• Additionally, some key/value pairs from the following files are used:
  • res/values/constants.xml – ids of some object detection panel types.
  • res/values/strings.xml – some strings for UI and image content descriptions.
  • res/values/styles.xml – panel style for object detection related panels.

• Fridge – a 3D model of a refrigerator, including PBR materials and two skeletal animations. One of the three “curated” objects, which is displayed when the user detects an object of this type in their surroundings.
• TV – a 3D model of a flatscreen television, including PBR materials and two skeletal animations. One of the three “curated” objects, which is displayed when the user detects an object of this type in their surroundings.
• Phone – a 3D model of a touchscreen smartphone, shaded with PBR materials. One of the three “curated” objects, which is displayed when the user detects an object of this type in their surroundings.
• WelcomePanel – a panel which is displayed when the user first loads the application. It initially shows a disclaimer (unless the user has already accepted the notice during a previous app session), and then shows a screen with a description of how to access the camera controls accompanied with an animated graphic demonstrating the action.
• CameraControlsPanel – a simple one button panel which is anchored to the user’s left wrist, and can be selected to toggle the object detection scanning on and off.
• HelpButtonPanel – a simple one button panel which is anchored to the user’s left wrist, and can be selected to view a dynamically spawned help panel providing a brief description of how to use the app, as well as an option to view the “curated” objects.

1. Passthrough Camera API
2. ML Kit Object Tracking
3. Llama 3.2 11B Vision Invocation

models/CuratedObject.kt
viewmodels/CuratedObjectInfoViewModel.kt
services/CuratedObjectHandler.kt
...
views/objectinfo/CuratedObjectImageCopyPanel.kt
views/objectinfo/CuratedObjectInfoScreen.kt
views/objectinfo/CuratedObjectTilesPanel.kt
...
res/xml/objects.xml

ecs/WristAttachedSystem.kt
...
app/src/main/components/WristAttached.xml

services/TipManager.kt
...
views/tips/FindObjectsScreen.kt
views/tips/HelpScreen.kt
views/tips/NoObjectsDetectedScreen.kt
views/tips/SelectObjectTipScreen.kt

app/src/shaders/9slice.frag
app/src/shaders/9slice.vert
app/src/shaders/customPanel.frag

• Meta’s Spatial SDK
• Meta Spatial UI Set – for Jetpack Compose UI styling and Horizon OS icons and typography.
• Google ML Kit Object Detection Vision API – for on-device object detection and tracking in the device live camera feed.
  • (optional) Google Mediapipe can be used in place of ML Kit for object detection, but this implementation currently does not support persistent tracking across video frames.
  • (optional) Open CV can be used in place of ML Kit for object detection, but this implementation currently does not support persistent tracking across video frames.
• AWS Kotlin SDK – for integrating AWS Bedrock for Llama 3.2 invocation.
• gson – for deserializing JSON response objects from AWS Bedrock.
• Jetpack Compose - for building modern Android UI with Material Design support.
• compose-markdown – for displaying markdown-formatted text in Jetpack Compose.