Virtualismo Song Avatar 3D

What it does

Our application uses information from songs that have been processed using Google Gemini to extract the text and separate it into paragraphs. The purpose of this separation is to associate possible feelings with each paragraph and use a 3D Avatar to synchronize the poses and animations of this Avatar with the audio of the song.
Song Processing
Text Extraction: We use Google Gemini to process the songs and extract the full text. The audio files of the songs, which can be in WAV or MP3 format, are uploaded to the application. Paragraph Separation: The extracted text is divided into individual paragraphs.
Sentiment Analysis: Each paragraph is analyzed to identify and associate possible sentiments.
API Exposure
An API is exposed so that the stored data can be consumed by other applications. This API provides access to song information, including the text of the paragraphs and the associated sentiments.
Example Application in JavaScript
API Consumption: The example application, developed in JavaScript, consumes the API to retrieve the songs data .
Audio Playback: The application allows for the playback of the song's audio.
Association with 3D Avatar: The recorded sentiments for each song paragraph are associated with an animated 3D avatar.
Sentiment Expression: The 3D avatar can express the sentiments through poses and facial expressions, providing a visual experience that complements the music.