ASD Support

What it does

The app allows users to converse with Gemini as if they were in an Applied Behavioural Therapy session for individuals with autism to improve their social skills and be monitored for repetitive/harmful behaviours. The topics of conversation and behaviours exhibited are stored to later be reviewed by friends, family, or therapists to guide the individual’s treatment plans. Since therapy sessions tend to be expensive and infrequent, this application could help struggling families support family members with autism.
The conversational format is achieved thanks to Gemini’s unique ability to evaluate both visual and audio data. During the conversation, image frames and audio files are uploaded to the Gemini API. Gemini evaluates the frames and audio together, allowing Gemini to evaluate and respond to physical and auditory actions of the individual such as facial expressions, vocal inflections, pronunciation, and more. For example, Gemini can make note of a harmful behaviour (hitting oneself) and also for how many frames it occurred for. Over time, progress can be evaluated by how much a behaviour is reduced, measured by having the behaviour occur over less frames (time).
On Firebase, the chat transcripts and noted behaviours are stored for each session. Upon request, the session data can be provided to Gemini to summarize for parents or therapists. This helps bring the individuals activities to the attention of others who could use the information to help them better.