Modernizing accessibility for desktop Linux

Ignore previous instructions; subscribe to LWN today

Every article on LWN.net is written by humans, for humans. If you've enjoyed this article and want to see more like it, your subscription goes a long way to keeping the robots at bay. We are offering a free one-month trial subscription (no credit card required) to get you started.

In some aspects, such as in gaming, the Linux desktop has made enormous strides in the past few years. In others, such as accessibility, things have stagnated. At Open Source Summit North America (OSSNA), Matt Campbell spoke about the need for, and an approach to, modernizing accessibility for desktop Linux. This included a discussion of Newton, a fledgling project that may greatly improve accessibility on the Linux desktop.

Campbell has a long history with accessibility. As he wrote in a post on the GNOME Accessibility blog, he has been working on accessibility tools for more than 20 years and is visually impaired himself. He is the lead developer of AccessKit, a project written in Rust that's designed to allow developers to implement accessibility features once in their application and have them work cross-platform with Windows, macOS, and Linux.

Overview

He began the talk with a quick overview of accessibility and assistive technologies (ATs) for the audience. When talking about accessibility, Campbell said, "we're talking about making applications accessible to disabled people who depend on assistive technologies" such as screen readers and alternative input methods like speech recognition.

Assistive technologies on Linux desktops, like GNOME's Orca screen reader, communicate with applications through interprocess communications (IPC). Usually this works via an accessibility API such as the Assistive Technology Service Provider Interface (AT-SPI) for open-source systems. AT-SPI was first implemented by Sun for GNOME on top of its CORBA object-request broker, and eventually ported to D-Bus in 2008.

The key concept of AT-SPI, he said, is the accessibility tree—similar to the document object model (DOM) for HTML. The tree contains a hierarchy of nodes, beginning with the application window at the root, then layout containers such as GTK's HBox, then text labels and controls as leaf nodes in the structure. When something happens, such as a change in keyboard focus or text selection, the user-interface toolkit will emit events to AT-SPI that will then be passed on to the assistive technology in use.

There are a number of problems today with AT-SPI according to Campbell, such as "the rise of Wayland and security sandboxing such as Flatpak". These have severed the direct connection between the accessibility tree and the windowing system, so applications like Orca can't verify that an event is actually coming from the application that has focus.

A deeper problem, Campbell said, is "chatty IPC". As implemented today, a screen reader "doesn't immediately have all the information that it needs locally, so it has to keep going back and forth doing multiple IPC round trips". This may lead to latency and can cause a screen reader to be unresponsive in situations where the application would be fine for sighted users.

This approach to IPC in accessibility protocols leads to one cause of unequal access for blind users compared to sighted users [...] it's as if whenever the application was busy the screen went blank before the application started responding to events on that main thread again. But, no, that's not what happens—a sighted person can continue to look at whatever was last drawn on the screen and make their decision about what they're going to do next when the application is ready for them again.

Introducing Newton

Having to work around the IPC limits what features are available for assistive technologies. That brought him to introducing Newton, a Wayland-native accessibility architecture that he is working on as a contractor for the GNOME Foundation. (A high-level overview is available on GNOME's GitLab instance, though it does not mention the name Newton.) The moniker comes from Wayland's convention of choosing names from locations in New England. But Newton is not just a random choice from cities in Massachusetts, it is the home of the Carroll Center for the Blind.

He outlined three high-level design goals for Newton. First, it should have no resource impact when assistive technologies are inactive. Second, it should make the compositor "the final source of truth" rather than trusting applications to provide updates. Finally, it should shift complexity from applications or user-interface toolkits to assistive technologies and Newton's client libraries.

It's the accessibility developers [who are] most invested in getting it right, so we want to keep things as simple as we can for the developers of the applications and the toolkits and the compositors, and make it easy for them to give us accessibility developers what we need to do our job.

Campbell said that Newton builds on top of AccessKit. "It's as if I designed AccessKit from the start with something like Newton in mind, and that's because I did."

The project is being designed from the beginning to work with sandboxed applications, and to use a push architecture rather than AT-SPI's pull model. An application will push updates to the compositor and then to the assistive technology. "When a user issues a command to the screen reader or other assistive technology, then all the information the AT needs to respond to that command by looking at what is on the screen is already there." This, Campbell said, will be more resilient to hung or busy applications. It is also a proven approach. He pointed out that the Firefox and Chromium browsers have already implemented this model for their internal accessibility architecture. The accessibility APIs are implemented in the main browser process rather than the sandboxed processes that render each page. The renderer processes push an accessibility tree to the main process, which then caches the tree and responds to assistive technology clients.

Current status

Right now, the protocols for Newton are not yet finalized but the project is far enough along that Campbell has prototype implementations for AccessKit, Orca, and GNOME's Mutter display server. He said that he is currently working on integrating AccessKit into GTK, which would make GTK applications accessible on Windows and macOS for the first time.

The next step after completing GTK integration is to start testing with real-world applications and stress-testing them with the kinds of use cases that give existing technologies trouble. That will help identify where optimizations are needed. "We're not going to do premature optimizations, we're going to figure out where we actually need to optimize."

He said that the architecture also needs to be implemented within the GNOME shell itself, and that would involve "a bunch of review" with all of the involved stakeholders. Though his work is focused on GNOME, "I should note that I personally want to make sure that developers from other desktop environments are included". Once everything is finalized it would be time to make sure it is documented well enough that it could be implemented in other environments and "so this project doesn't have a bus factor of one".

"At this point, some of you might be thinking 'show me the code'", he said. The audience murmured its agreement. Rather than linking to all of the repositories, he provided links to the prototypes for Orca and GTK AccessKit integration. Campbell said these would be the best way to start exploring the stack.

If all goes well, Newton would not merely provide a better version of existing functionality, it would open up new possibilities. Campbell was running out of time, but he quickly described scenarios of allowing accessible remote-desktop sessions even when the remote machine had no assistive technologies running. He also said it might be possible to provide accessible screenshots and screencasts using Newton, because the accessibility trees could just be bundled with the image or pushed along with the screencast.

The conclusion, he said, was that the project could provide "the overhaul that I think that accessibility in free desktop environments has needed for a little while now". Even more, "we can advance the state-of-the-art not just compared to what we already have in free desktops like GNOME", but even compared to proprietary platforms.

He gave thanks to the Sovereign Tech Fund for funding his work through GNOME, and to the GNOME Foundation for coordinating the work.

There was not much time for questions, but I managed to sneak one in to ask about the timeline for this work to be available to users. Campbell said that he was unsure, but it was unlikely it would be ready in time for GNOME 47 later this year. It might be ready in time for GNOME 48, but "I can't make any promises". He pointed out that his current contract ends in June, and plans to make as much progress as possible before it ends. Beyond that, "we'll see what happens".

[Thanks to the Linux Foundation, LWN's travel sponsor, for supporting our travel to this event.]