SEP
8
2012

Accessibility in KDE 4.10 and beyond

The history

The first tries to join forces with GNOME to build up one common accessibility stack for our beloved desktops pre-date back to 10 years ago.

A year later the solution became more concrete and finally made it as the QAccessible framework into Qt 4.0 in ~2004.

QAccessible is, just like Qt itself, cross platform. That means it uses the a11y-infrastructure available at Windows when Qt/KDE applications run there and whatever is available at other platforms. Under Linux the situation was a bit more difficult. That time where years of big shifts, ranging from the corba/dcop merger that resulted in dbus up to the work towards KDE4. It also directly affected the work happened on our common goal for a shared accessibility stack for GNOME and KDE.

Time passed, dbus got adopted all along, the GNOME atspi accessibility stack moved to dbus, KDE4 came out and Qt4 and with it QAccessible improved future during all that time. Somewhere in 2008 a few big organizations invested to finally bring Qt-Linux to atspi and qtatspi was born.

Later on atspi, that is to be more precise atspi2 which was atspi1 on dbus rather then corba, became a Linux Foundation project and finally the official accessibility stack for Linux. Our common goal was close except that atspi2 itself was still not there. Development went on, APIs got large refactorings, the qtatspi bridge became obsolete, years passed.

But while those atspi2-years passed QAccessible2 was born and made it into Qt. It never became public API, its undocumented but it's there and just works. Also qtatspi got a redux and was reborn. Future heavy investment was made. This time from Nokia (thanks!) which continued the heavy investment on Qt accessibility done by Trolltech all the years before.

Today we have a really great accessibility-framework in Qt4 and even future improved in upcoming Qt5. Its still undocumented but it even works better then it did before. In fact today Qt comes with QAccessible1 and QAccessible2, two accessibility frameworks. QAccessible1 was never removed or official replaced cause Qt4 was not allowed to change public API. That was also the reason QAccessible2 never became official public API cause once done you cannot do any binary incompatible changes any longer. Afaik QAccessible2 is to be scheduled to become official Qt API with 5.1/5.2. We will see. Meanwhile the API's are there and proven since years to work.

This days qtatspi is developed in the KDE repositories. It works great on Qt4 and Qt5 and was meanwhile merged into Qt5 (afaik). That means Qt 5.0 will give us, beside drastic improved accessibility-support for QML, also support for the common atspi Linux accessibility stack out of the box.

10 years of history, small steps but always on move, one step back and two steps forward and finally the vision of a shared accessibility stack is there.

The fruits



The left window is gedit, the GNOME text-editor written using the GTK toolkit. The right window is our new accessibility inspector (no proper name yet) that we wrote to test the new libkdeaccessibilityclient-lib (name going to change too). While qtatspi bridges Qt-applications to atspi our libdeaccessibilityclient bridges tools that make actual use of the accessibility-functionality (like magnifiers and screenreaders) to atspi. In the screenshot above our accessibility inspector is accessible itself displayed in the tree as "TestApp" while also such a tool cause it uses atspi to display various informations about itself and gedit.

In the tree I selected the "New" tree item. That equals to the File=>New menu-item in gedit. On the right of the tree are some (not all) of the properties those menu-item has displayed. Also the menu-item offers an action to execute it. That is the popup with the "click" menu-item in it. When I click that menu-item then gedit would open a new blank document tab.

At the bottom is the Events-view which displays all kind of events the application received/processed/send. That includes things like Window activated/deactivated/minimized/etc, like menu-item got focused or like text cursor moved/text changed and more.



The screenshot above shows the accessibility inspector's boundaries view. In the treeview I selected the item that is responsible to draw the "File Open" toolbar button in gedit. In the boundary view a screenshot of the application is drawn, the different regions are colored and a red centangle is drawn around the currenly selected item.

The whole application/atspi/tool communication works like:
Qt/KDE application <=> qtatspi <=> atspi <=> libdeaccessibilityclient <=> screenreader/magnifier/etc.
Note that on both ends can be Qt/GTK apps. That means Qt/KDE applications are also accessible using tools like the Orca screenreader and Accerciser just like gedit is accessible using our inspector tool. Interoperability pure and a prime example of great cooperation among KDE and GNOME.

Beside the nice accessibility inspector tool there was also very great work done on the magnification front:
http://amanonit.blogspot.com/2012/06/gsoc-focus-tracking-working-in-kwin.html
http://amanonit.blogspot.com/2012/08/summer-of-accessibility.html
The accessibility tool story just started and I am sure there is a bright future for atspi.

The possibilities

atspi does not only allow us to deeply inspect an application but also to interact with it. Someone can fetch the text displayed in an edit-field and set/manipulate the text, can fetch the actions an application offers and execute them, can trigger buttons, open menus, click menu-items, change the focus and so on.

Imagine the possibilities this opens far beyond "boring accessibility". Full remote-control of applications, automatisation, testing, implementation of own UI's. Yes, someone could go on and write an application that provides for all the applications out there a common interface through a single controller-app pffering the same workflow, look & feel without actually even have to deal with the real user-interface the application ships with.

The limitations

First the good news: *ALL* Qt and KDE widgets are accessible out of the box. Usually if you stick with the Qt and KDE default widgets your application is already fully accessible :-)

Now the bad news: The moment you wrote your own widget or maybe even have central informations or controls in a canvas and draw all yourself using QPainter/OpenGL the moment your widget/canvas/scene is a blackbox and not accessible at all. Another problem currently is QGraphicsView but the plan is to address that more sooner then later. QML is accessible and there are (afaik unfinished) patches for QtWebKit too.

The other big limitation is, that QAccessible2 is currently undocumented API in Qt. While the plan is to make it official public API soon its not yet. Things are stable and just work (TM) but that means the information how to use them isn't easy to discover yet.

More resources

There are some examples that show you how to make own widgets/canvases accessible up at:
https://projects.kde.org/projects/kde/kde-baseapps/kate/repository/revisions/master/entry/part/view/kateviewaccessible.h
https://projects.kde.org/projects/kde/kde-baseapps/konsole/repository/revisions/master/entry/src/TerminalDisplayAccessible.cpp
https://git.reviewboard.kde.org/r/105972/diff/#index_header

In Qt4 the relevant code that bridges all the widgets to QAccessible is at qt/src/plugins/accessible/. You can also grep for QAccessible in qt/src/ to see where and how else QAccessible is used in Qt.

qtatspi, that is the plugin that bridges QAccessible to atspi, is available at git://anongit.kde.org/qtatspi . Instructions are at http://community.kde.org/Accessibility/qt-atspi .

If you have questions please join us in IRC at #kde-accessibility on freenode (use e.g. Konversation).

Thanks goes to:
Harald Fernengel who started that and drived it forward all the years,
Frederik Gladhorn who took qtatspi over, made it into a winner.
SUN, Trolltech and Nokia for all the investment over the years.
Mozilla for the initial qtatspi sponsoring.
KDAB for allowing me to extend expertise in that field.

Comments

The remote control / testing stuff has already been done: https://fedorahosted.org/dogtail/.


By Kevin Kofler at Mon, 09/10/2012 - 01:29

Thanks for the pointer. Great to know/see :)


By Sebastian Sauer at Mon, 09/10/2012 - 08:28