Latte: Use-Case and Assistive-Service Driven Automated Accessibility Testing Framework for Android

Navid Salehnamadi Ziyao He S. Malek

19 April 2023

Billions of people use smartphones on a daily basis, including 15% of the world’s population with disabilities. Mobile platforms encourage developers to manually assess their apps’ accessibility in the way disabled users interact with phones, i.e., through Assistive Technologies (AT) like screen readers. However, most developers only test their apps with touch gestures and do not have enough knowledge to use AT properly. Moreover, automated accessibility testing tools typically do not consider AT. This paper introduces a record-and-replay technique that records the developers’ touch interactions, replays the same actions with an AT, and generates a visualized report of various ways of interacting with the app using ATs. Empirical evaluation of this technique on real-world apps revealed that while user study is the most reliable way of assessing accessibility, our technique can aid developers in detecting complex accessibility issues at different stages of development.

Maryam Taeb Ruijia Cheng E. Schoop + 3 lainnya

3 Oktober 2023

Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs. However, to our knowledge, no one has yet explored the use of LLMs in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of a natural language based accessibility testing workflow, starting with a formative study. From this we build a system that takes a manual accessibility test instruction in natural language (e.g., “Search for a show in VoiceOver”) as input and uses an LLM combined with pixel-based UI Understanding models to execute the test and produce a chaptered, navigable video. In each video, to help QA testers, we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops). We evaluate this system through a 10-participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and performed tests similarly to how they would manually test the features. The study also reveals insights for future work on using LLMs for accessibility testing.

Letícia Seixas Pereira Maria Matos Carlos Duarte

11 Mei 2024

With the ubiquitous use of mobile applications, it is paramount that they are accessible, so they can empower all users, including those with different needs. Determining if an app is accessible implies conducting an accessibility evaluation. While accessibility evaluations have been thoroughly studied in the web domain, there are still many open questions when evaluating mobile applications. This paper investigates mobile accessibility evaluation methodologies. We conducted four studies, including an examination of accessibility reports from European Member-states, interviews with accessibility experts, manual evaluations, and usability tests involving users. Our investigations have uncovered significant limitations in current evaluation methods, suggesting that the absence of authoritative guidelines and standards, similar to what exists for the web, but tailored specifically to mobile devices, hampers the effectiveness of accessibility evaluation and monitoring activities. Based on our findings, we present a set of recommendations aimed at improving the evaluation methodologies for assessing mobile applications’ accessibility.

Alberto Dumont Alves Oliveira D. M. Eler Wajdi Aljedaani + 3 lainnya

19 April 2023

Accessibility reviews collected from app stores may contain valuable information for improving apps accessibility. Recent studies have presented insightful information on accessibility reviews, but they were based on small datasets and focused on general accessibility concerns. In this paper, we analyzed accessibility reviews that report issues affecting users with visual disabilities or conditions. Such reviews were identified based on selection criteria applied over 179,519,598 reviews of popular apps on the Google Play Store. Our results show that only 0,003% of user reviews mention visual disabilities or conditions; accessibility reviews are associated with 36 visual disabilities or eye conditions; many users do not give precise feedback and refer to their disability using generic terms; accessibility reviews can be grouped into general topics of concerns related to different types of disabilities; and positive reviews are generally associated with high scores and negative feedback with lower scores.

J. Bhatia Dhruv Nagpal Parthasarathy P D + 2 lainnya

12 Oktober 2022

The growing interest in accessible software reflects in computing educators' and education researchers' efforts to include accessibility in core computing education. We integrated accessibility in a junior/senior-level Android app development course at a large private university in India. The course introduced three accessibility-related topics using various interventions: Accessibility Awareness (a guest lecture by a legal expert), Technical Knowledge (lectures on Android accessibility guidelines and testing practices and graded components for implementing accessibility in programming assignments), and Empathy (an activity that required students to blindfold themselves and interact with their phones using a screen-reader). We evaluated their impact on student learning using three instruments: (A) A pre/post-course questionnaire, (B) Reflective questions on each of the four programming assignments, and (C) Midterm and Final exam questions. Our findings demonstrate that: (A) significantly more (p <.05) students considered disabilities when designing an app after taking this course, (B) many students developed empathy towards the challenges persons with disabilities face while using inaccessible apps, and (C) all students could correctly identify at least one accessibility issue in the user interface of a real-world app given its screenshot and 90% of them could provide a correct solution to fix it.