How to (re)-position the virtual objects in the real world in an Augmented Reality experience – while still having an interactive scene? Elegantly guide your users through the placement process.
The official AR tutorial from Amazon contains a simple script: by tapping anywhere in the scene, it instantly moves the objects to that position. However, for the Digital Healthcare Explained app, I needed a more flexible behavior:
- Activate placement mode by tapping on a specific object in the 3D scene. In this case, I decided that tapping the host avatar triggers placement mode.
- The host then explains what to do: tapping on another surface moves the host and related objects. Guide users through the process. The Sumerian hosts are ideal to explain the process.
- The user taps on a real-world surface in the AR scene.
- Next, the scene contents move, the anchor updates and the host confirms.
New ES6 Based Scripting
Additionally, Amazon Sumerian is evolving its scripting language. A major upgrade to ES6 is underway. It’s fully based on classes and fits better into the actions and state machines used in other places of Sumerian. The new APIs are still marked as “Preview”. However, the old APIs are already called “Legacy” or “Old Script Format”.
While documentation for the new Sumerian Engine APIs directly is already available, it’s very brief and doesn’t contain many examples. The official tutorials are still based on the legacy API.
I decided to re-write the script using the new APIs. It involves calling a lot of internal parts of Sumerian. Thus, it’s a lot more complex than all other examples for the new API currently out there. However, it’s interesting to dig more into the internals of how a modern, web-based AR environment works.
Scripts and Variables in Sumerian
Before we dive into AR specific topics, we need to take a closer look at the new scripting system in Amazon Sumerian.
Any entity within a Sumerian scene can have one or more script components attached. These have certain pre-defined methods according to the object lifecycle. The most important ones:
- start(ctx): called when play mode starts
- UpdateAction(): define a callback that gets executed each frame, if necessary
- ctx.onStop(): callback, executed when play mode stops
In most of these life-cycle methods, ctx is directly or indirectly invovled. This refers to the Context Object in Sumerian. Scripts can use it to start other actions as well as to send and receive messages.
These messages can be global events (world events). In that case, they connect different parts of your scene. For example, the host informs our script that the user started the AR placement mode. Once the user finished, our script sends that info back to the rest of the scene. The scene can resume its normal functionality.
Alternatively, the ctx.world can also simply store shared values that multiple parts of your scene can access, update or subscribe to. For our scenario, a global state is useful when we’re in placement mode. During that time, other interactions in the world are inactive. Only when the user taps somewhere again to finish placing the scene entities, the other interactivity is restored.
Entity Hierarchy for AR Object Placement
Let’s create a script that is easily to re-use. As such, we want it to be configurable as to which entity should own the AR anchor. In case we don’t want to set it, the script defaults to the entity it’s attached to.
Why do we need that? In our scenario, we place the new “Anchor Positioning” script on the entity we want to own the AR placement – the host. After all, the host guides the user through the process, and therefore manages the interactions and decides when placement is possible at all.
However, the AR anchor itself attaches to the parent entity of the host. After all, we don’t only want to move the host in this scenario, but also several other related objects in the world. In my demo scenario, the user interface is always next to the host.
This is what our entity hierarchy looks like:
Create an empty entity and call it “PlaceableWorld”. Make all scene objects you would like to move together children of that entity. In our case, this is the host (“Grace”), as well as other 3D models around it – like some buttons (“Help”, “Privacy Policy Icon”, etc.).
The “Anchor Positioning” script is then attached to the host (“Grace”), as placement mode is triggered by tapping the host. An alternative could be to create a button in the UI to start placement mode.
Next, select your host. Add a “Script” section in the inspector pane on the right if it’s not there already. Then, create a new script based on “Custom (Preview Format)” and call it “Anchor Positioning”.
Anchor Positioning Script Structure
First of all, you need to import Sumerian scripts that we will access from our script. As promised, our code will dive deeper into Sumerian APIs. Thus, we need both the “normal” Sumerian API (imported as s), as well as internal APIs (imported as si).
In addition, we extend our AnchorPositioning script from s.Action. This would for example allow it to be embedded into state machines.
Script Properties
How to make properties of your script accessible in the web-based Sumerian editor? Simply add static “PROPERTIES” to your class. They contain the definition of all parameters. Most importantly:
- [name]: the ‘internal’ name how you refer to the parameter from your script.
- description: the ‘external’ name or description visible in the Sumerian web editor.
- type: one of Sumerian’s types. This also defines the user interface shown for this property in the editor. For example, the UI could be a number editor, or like in our case a selector for a target entity in the scene. Additionally, you could further customize the UI for some types – for example by setting minimum and maximum values.
For our “Anchor Placement” script, we want the user to (optionally) define the entity that owns the AR anchor. Copy the following property definition into your script file – for example at the top of the class, outside of any function.
Script Configuration
Back to the Sumerian editor, you now see the configurable parameter in the inspector panel of your host. Drag and drop the “Placeable World” entity from the hierarchy panel on the left onto the script argument. In the end, it should look like this:
Next up…
Coming up next, we will take a look at how Amazon Sumerian handles Augmented Reality anchors. These are provided by the native AR system (ARCore / ARKit). Sumerian scenes run entirely in web browsers – which don’t support AR yet. Therefore, some wrapper code of Sumerian glues the native world together with its web environment.