Introduction
One of the interesting things we can do with virtual reality is to decorate the real world with digital data. Imagine transforming your living space with just a few taps on your screen. With augmented reality tools like ARKit in visionOS, this is not only possible but becoming easier to do. From knowing where a wall is to getting the shape of a table, we can handle the shape of the world to make what we want.
Sessions, providers and anchors
ARKit, Apple’s augmented reality framework, offers three fundamental features that make integrating real-world data into your applications possible: sessions, providers, and anchors. Let’s break down what each of these components does and how they interact with each other.
A session in ARKit is responsible for managing the flow of data between your application and the AR environment. When you start a session, you specify a set of data providers that will gather the necessary information from the environment. These providers continually retrieve and update data, enabling your app to understand and interact with the real world in real-time. Sessions must be run in a full-space environment to function correctly, ensuring the AR experience covers the intended physical space.
Data providers are responsible for collecting specific types of information from the real world. Each provider focuses on a particular data type, such as plane detection or world tracking. By observing a data provider, you can access real-time updates as the environment changes. For example, the PlaneDetectionProvider detects flat surfaces like walls and tables, while the WorldTrackingProvider tracks the device’s position and orientation in space.
An anchor in ARKit is a reference point in the real world that helps position virtual objects accurately. Each anchor has a unique identifier and can be of different types depending on your application’s needs. For example, PlaneAnchors represent detected surfaces like floors or walls, while WorldAnchors mark specific points in space that remain consistent across app sessions.
The PlaneDetectionProvider scans your environment to identify flat surfaces, returning a set of PlaneAnchors. Each PlaneAnchor includes detailed information such as the plane’s geometry (size and shape) and its classification (e.g., wall, table, floor). This data is crucial for placing virtual objects accurately on these surfaces.
Similarly, the WorldTrackingProvider offers essential features for spatial awareness. It provides WorldAnchors, which mark fixed points in your environment, and tracks the device’s movement. This allows your app to persist the positions of virtual objects even when the app is restarted. Entities or other type of data are not automatically persisted as well, but that can be easily solved by persisting it manually, with a database or even a single json file saving the id of the anchor and the content related to it.
You can see other types of data providers in this documentation:
https://developer.apple.com/documentation/arkit/dataprovider#conforming-types
Here is an example of a simple session running, using the world tracking provider to retrieve the world anchors in our surroundings:
In the example above, we initialize a world tracking provider and an ARKitSession. Then we run the session when the RelityView creates and pass the provider with the data we want to retrieve (in this case, it’s just the world tracking provider, but we can pass more providers).
Then, we create a task to asynchronously iterate over our anchor updates, and do something for every type of event, for example if the anchor is removed, we can remove its associated data.
Decorate the world
Now that we know how anchors work, we can go to the decoration part.
Imagine you have an Entity of a painting, and you want to anchor it to your wall. Then, the first step is to place your entity there, and there are many ways to do it, you can add it to the space and drag it yourself, or even track a plane anchor with the PlaneAnchorProvider and search for a wall and then add your entity there. Once your entity is in its place, the next step is to add a world anchor for it. For that, you can do something like this:
This function, ‘attachEntityToWorldAnchor’, takes an Entity and uses its transform matrix to create a World Anchor. By matching the transform matrix, you ensure that the exact position, scale, and rotation of your entity are saved at a specific point in space. This way, even if the app restarts, the anchor remains in the same location, although the entity itself must be re-added programmatically, because as we mentioned earlier, world anchors are automatically persisted but not their entities. For that, you will need to save it yourself.
An example for that is using a json file, keeping track of your anchored entities with a map that contains the anchor id as the key, and for example your entity type as a value. We will use a String as a value that contain the entity model name:
To persist anchors and their associated entities, we use a dictionary called ‘worldAnchors’ that maps anchor IDs to entity model names. When an anchor is added, we update this dictionary. The ‘saveWorldAnchors’ function encodes this dictionary into a JSON string and saves it to a file, ensuring that the mapping is preserved between app sessions. Similarly, the ‘loadWorldAnchors’ function reads the JSON file upon app start and restores the dictionary, allowing the app to recreate the entities at their saved positions. This way is pretty simple to write our data, and it is also simple to load it when our app starts again:
With this function, we search for the json file, if it exists, we decode it into a map and assign it to our worldAnchors map, and there we have our world anchors associated with the entities we used before.
So now we can add an entity, create a world anchor for it, persist it and load it on application boot with those functions, but even with this the entities are not appearing again. That is why we need to create them and add them to our content.
For that, we need to listen to our anchor updates that we defined earlier, and check if we have any reference to them in our map, if we have, we use the string to create our entity and the anchor to get the transformation matrix, for example:
And this way, our entities should be recreated in the same place they were the time we added them.
Conclusion
In conclusion, ARKit provides us with multiple tools to get data from our environments and use it, we can manage to use some of those features to decorate indoor and outdoor spaces with entities and save them in memory to see them again every time we run the application.
References
https://developer.apple.com/videos/play/wwdc2023/10082/
https://developer.apple.com/documentation/arkit
https://developer.apple.com/documentation/visionos/tracking-points-in-world-space
https://developer.apple.com/documentation/visionos/placing-content-on-detected-planes
