What is it ?
A small experiment to learn how the Passthrough Camera API (Unity pca) works.
This allows a Mixed-Reality game on platforms like the Meta Quest to recognize and react to specific objects in the player’s environment, which could allow for example Toys-to-Life features without the need for NFC chips, or location-based features without the need for GPS.
I had to learn on the fly about a Model-Language in python while making it. I trained an AI to detect and recognize the nexus poster. I then spawned a “game level” when that poster is detected.
How did I make it ?
In Python:
- The team took several pictures of Nexus posters at different angles.
- I trained YoloV8 ML using those images.
- I iterated on it until the detection was good enough for a proof of concept.
Inside Unity:
- Combined two samples, “CameraToWorld” and “MultiObjectDetection”
- With the “CameraToWorld”, I captured a whole image from the camera. With the “MultiObjectDetection”, I found where the poster was located and cropped it from the image.
- Then I spawned that image at the poster location, moved that image near the player, and spawned the level.