SANT AR

What it does

Our app allows kids to assemble SCABO cardboard toys with 3D detailed instructions. Then, children can turn the cardboard toys into smart toys in AR with Gemini AI capabilities. Our app can detect and track 3D position of cardboard toys in realtime with our own 3D computer vision algorithm. We utilize Gemini 1.5 Pro capabilities including vision and text completion to become AI assistant for children in learning scientific subjects. Our 3D inventory system including more than 500 simulations in human anatomy, mechanics, and space models. We designed the 3D simulations by texturing the 3D models with different color for different parts of scientific objects to prevent AI-hallucination. Gemini AI recognizes the virtual simulations better and give better explanations for kids. Furthermore, we use the Gemini API to detect object in the scientific scenes. Given the image including the foreground 3D virtual models or real artifact and a question from users such as "Where is differential gear", Gemini AI object detection can detect 2D position of the parts. Then, we have 3D mapping algorithm to map the 2D position into the scientific object. Lastly, we encourage children to reuse their existing plastic toys in the AR toy stories mode. Gemini API is used to generate the sci-fi stories between virtual scientists and the plastic toy. Children can both learn and talk with virtual 3D simulations or their own toys in scientific scope. All 3D simulations are free & on the fly.