Textbooks Are All You Need (TAAYN-chan)

What it does

Textbooks Are All You Need (Meet Taayn-chan) is an innovative ARCore-enhanced interactive textbook app that revolutionizes learning by bringing educational content to life. The app combines augmented reality, the Gemini API, and advanced voice technology to create an immersive, personalized, and humorous learning environment.
At its core, the app features Taayn-chan, an AR avatar who serves as a friendly, intelligent tutor and guide. This avatar appears in the user's physical space through their device's camera, creating a sense of presence and engagement. When users ask Taayn-chan questions about the textbook content, he provides personalized answers with AR figures that pop out of the book, making the learning experience more engaging and interactive.
The app's key feature is its integration with the Gemini API, which powers Taayn-chan's AI capabilities. Speech-to-text is performed locally on the device using whisper.cpp, while text-to-speech is provided by the ElevenLabs API, ensuring smooth and natural conversations.
Taayn-chan's personality and knowledge are based on a reference context cache from Gemini containing his instructions and the textbook contents. This allows him to maintain consistent behavior and accurate information delivery.
Currently a prototype, TAAYN uses textbooks and content that are either open-source or in the public domain, all used with appropriate permissions.