Static code analysis cannot detect real-time interaction issues faced by users with disabilities. We propose a multi-modal Artificial Intelligence (AI) agent framework that simulates interactions of users with visual impairments without code access. The agent would closely simulate the experience of these users by interacting with web interfaces using the same modalities available to them, primarily keyboard navigation and screen readers. The agent perceives the interface through perceptual filters that mimic conditions including glaucoma and myopia, and handles both the altered visual input and audio output from screen readers. This approach aims to replicate the real-world constraints and strategies of users with disabilities, enabling more realistic evaluation, with the objective of identifying, locating and repairing web accessibility issues. We suggest a framework to evaluate how such filters affect user behavior, task success, and User Interface (UI) usability. Our approach aims to uncover visual accessibility flaws that become apparent under impaired perception.

In the field of human-computer interaction (HCI), alternative interaction methods are becoming increasingly popular and commercially available. From this opportunity came EyeNav, a system that combines eye-tracking and natural language processing (NLP) to enhance accessibility and enable automated test generation. The integration of these technologies for intuitive web interaction, using pointer control via gaze and natural language processing for interpreting user intentions, also presents a record-and-replay module for generating automated test scripts. Envisioned for users with motor disabilities, developers, usability testers, and general users interested in exploring novel multimodal web interactions. The ultimate goal is to demonstrate that this tool can be used not only as a possible assistive technology but also as an innovative approach to software testing. The tool is available publicly at https://thesoftwaredesignlab.github.io/EyeNav/, accompanied with a demonstration video https://thesoftwaredesignlab.github.io/EyeNav/video.html