University of Houston-Downtown
Presenter: John M Esparragoza, University of Houston-Downtown
Project: Utilizing AI to Support in Communication with ASD Individuals
Description of the project: This project investigates how artificial intelligence, specifically computer vision (CV) and natural language processing (NLP), can support communication for individuals with autism spectrum disorder (ASD). The system generates simplified, meaningful narratives from user-uploaded images and adapts them to the user’s communication level using a Most-to-Least (MTL) prompting strategy drawn from applied behavior analysis (ABA). The resulting prototype, built in Streamlit and integrated with MC-LLaVA and text-to-speech (TTS), provides an accessible, low-complexity interface suitable for individuals who experience sensory or linguistic overload.
Early testing demonstrated the feasibility of using a lightweight multimodal model (MC-LLaVA-3B) to produce concise captions with fewer hallucinations than larger alternatives like BLIP-2. The prototype successfully allows image upload, caption generation, audio playback, and prompt-fading controls, offering evidence that AI-assisted narrative support can be implemented in an affordable and user-adaptive way.
The initiative is cost-effective due to its reliance on open-source tools and compact pre-trained models, minimizing the need for large annotated datasets or expensive infrastructure. Key lessons learned include the importance of realistic scoping (image-only input), selecting models that balance performance and computational cost, and prioritizing accessibility and user adaptability during design.
Biography: I’m an Undergraduate Computer Science student at the University of Houston-Downtown (UHD) with a growing interest in machine learning/AI and applied research. I’ve had the chance to work on edge computing projects through research experiences at Texas State University, exploring computer vision and natural language processing. I enjoy learning how tech can solve real-world problems, and when I’m not coding or studying, you’ll probably find me at the gym, binging shows/anime, or catching up on sleep.