Programmatically providing direct access to user interface elements of an application program

Abstract

An architecture is provided that enables an accessibility aid to directly access and manipulate user interface elements of an application program programmatically. Moreover, such access and manipulation occurs in an application-independent manner, so that an accessibility aid utilizing the architecture can access any application program that conforms to the architecture, without the accessibility aid needing any prior knowledge of the application program or its user interface elements. User interface elements typically have both a visual representation displayed on the video display and an implementation, which is the code and data implementing the user interface element. The architecture provides an accessibility aid with direct access to the implementation of user interface elements, thus enabling the accessibility aid to both examine various characteristics of the user interface element and manipulate these characteristics, which may affect its visual representation.

Claims

What is claimed is:

1. A method performed by an application program in a computer system for providing direct access to user interface elements of an application program, comprising the steps of:

selecting, by a client application, a programmatic access component for use by the client application;

receiving, from the client application, a programmatic request identifying a user interface element to access via the programmatic access component;

accessing the identified user interface element on behalf of the client application responsive to the received programmatic request;

attaching a description to the identified user interface element;

requesting the description by the programmatic request; and

returning the description to the client application by the step of directly accessing the identified user interface element; and

conveying said description of said user interface element to a user.

2. The method of claim 1, wherein the step of selecting a programmatic access component includes selecting an interface containing at least one function member that is utilized by the client application to directly access at least one of the user interface elements of the application program.

3. The method of claim 1, further comprising the steps of:

including a default action in the identified user interface element;

requesting by the programmatic request that the application program perform the default action on the identified user interface element; and

performing the default action on the identified user interface element by the step of directly accessing the identified user interface element.

4. The method of claim 1, further comprising the steps of:

connecting the user interface element to a plurality of relations;

requesting one of the plurality of relations by the programmatic request;

accessing a related user interface element that is related to the identified user interface element via the requested relation; and

returning a reference to the related identified user interface element to the client application.

5. The method of claim 1, wherein the programmatic request selects the identified user interface element.

6. The method of claim 1, further comprising the steps of:

attaching a help string to the identified user interface element that provides useful information to the user;

requesting the help string of the identified user interface element by the programmatic request; and

returning the help string to the client in the step of directly accessing.

7. The method of claim 1, further comprising the steps of:

attaching an associated help topic to the identified user interface element that provides useful information to the user;

requesting an identifier of the associated help topic of the identified user interface element by the programmatic request; and

returning the identifier of the associated help topic to the client in the step of directly accessing.

8. The method of claim 1, further comprising the steps of:

attaching an invocable response to a user depression of a key on a keyboard input device;

requesting an indication of the key by the programmatic request; and

returning the indication of the key to the client application in the step of directly accessing.

9. The method of claim 1, further comprising the steps of:

associating a name to the identified user interface element;

requesting the name associated to the identified user interface element by the programmatic request; and

returning the name to the client application in the step of directly accessing.

10. The method of claim 1, further comprising the steps of:

associating one of the user interface elements with a parent user interface element having a parent/child relationship with a child user interface element;

supporting the parent user interface element with an IDispatch interface;

associating the identified user interface element as the child user interface element;

requesting a reference to the IDispatch interface of the parent by the programmatic request;

accessing the reference to the IDispatch interface of the parent user interface element by the step of directly accessing; and

returning the reference to the IDispatch interface to the client in the step of directly accessing.

11. The method of claim 1, further comprising the steps of:

linking a value to the identified user interface element;

providing a new value for the user interface element by the programmatic request; and

setting the value of the user interface element to the new value by the directly accessing step.

12. The method of claim 1, further comprising receiving, by the client application, a notification when the user interface element changes status.

13. A method for accessing user interface elements of a first application program performed by a client application using a computer system, comprising the steps of:

requesting, by the client application, a first programmatic access component from the first application program;

receiving the first programmatic access component;

utilizing the first programmatic access component to directly and programmatically access at least one of the user interface elements on behalf of the client application;

conveying a description of said at least one of the user interface elements to a user.

14. The method of claim 13, wherein the first programmatic access component is an interface containing at least one function member that provides access to the identified user interface element.

15. The method of claim 13, further comprising the steps of:

requesting a second application program having a second programmatic access component, where the second application program is substantially different than the first application program and the second programmatic access component has a definition for utilization that is equivalent to the definition of the first programmatic access component;

receiving the second programmatic access component from the second application program; and

utilizing the second programmatic access component in a substantially same manner as the first programmatic access component to directly and programmatically access at least one of the user interface elements of the second application program.

16. The method of claim 15, further comprising the step of:

requesting that the second application program perform a default action on the identified user interface element.

17. The method of claim 13, further comprising the step of:

referencing the identified user interface element that supports an IDispatch interface.

18. The method of claim 13, further comprising the steps of:

connecting the user interface elements to a plurality of relations;

requesting one of the relations by the first programmatic access component;

accessing a related user interface element that is related to the at least one user interface elements via the indicated relation; and

receiving a reference to the related user interface element.

19. The method of claim 13, further comprising the step of:

selecting the at least one user interface elements by utilizing the first programmatic access component.

20. The method of claim 13, further comprising the steps of:

attaching a help string to the at least one user interface elements that provides useful information to the user;

requesting the help string of the at least one user interface elements by the first programmatic access component; and

receiving the help string of the identified user interface element from the first application program.

21. The method of claim 13, further comprising the steps of:

attaching a help topic to the at least one user interface elements;

requesting the help topic of the at least one user interface element by the first programmatic access component; and

receiving the identifier of the help topic from the first application program.

22. The method of claim 13, further comprising the steps of:

associating a name to the at least one user interface elements;

requesting the name associated to the at least one user interface elements by the first programmatic access component; and

receiving the name from the first application program.

23. The method of claim 13, further comprising the steps of:

associating one of the user interface elements with a parent user interface element having a parent/child relationship with a child user interface element;

supporting the parent user interface element with an IDispatch interface;

associating the child user interface element with the parent user interface elements;

requesting a reference to the IDispatch interface of the parent user interface element by the first programmatic access component;

accessing the reference to the IDispatch interface of the parent user interface element by the step of directly accessing; and

receiving the reference to the IDispatch interface of the parent user interface element from the first application program.

24. The method of claim 13, further comprising the steps of:

linking a value to the at least one user interface elements;

providing a new value for the at least one user interface element by the first programmatic access component; and

setting the value of the at least one user interface element to the new value.

25. The method of claim 13, further comprising receiving, by the client application, a notification when the user interface element changes status.

26. A method for accessing user interface elements of a first application program by a client application using a computer system, comprising the steps of:

requesting, by the client application, a reference to an interface supported by the first application program, the interface comprising at least one function member that provides access to at least one of the user interface elements;

receiving the reference to the interface;

invoking the function member to directly access and manipulate the user interface element in a second application independently such that access and manipulation of the second application user interface elements support the interface in a same manner; and

receiving, by the client application, a notification when one of the user interface elements changes status.

27. A computer system comprising:

a video display displaying user interface elements;

a memory, further comprising:

an application program for displaying the user interface elements on the video display, and for providing direct access to an implementation of the user interface elements by selecting an interface containing function members that each access the implementation of the user interface elements;

an accessibility aid for invoking the function members on the interface to provide a user with access to the implementation of the user interface elements; and

an operating system for providing the accessibility aid with access to the interface of the application program and for conveying a description of said user interface element to the user; and

a processor for running the application program, the operating system, and the accessibility aid.

28. A computer-readable medium containing instructions for controlling a computer system to perform a method for accessing a user interface element of a computer program, the user interface element having a class definition that is able to be instantiated into an object, the method comprising the steps of:

requesting, by a client application, a reference to an interface supported by the first application program, the interface comprising at least one function member that provides access to at least one of the user interface elements;

receiving the reference to the interface;

invoking the function member to directly access and manipulate the user interface element in a second application independently such that access and manipulation of the second application user interface elements support the interface in a same manner; and

receiving, by the client application, a notification when one of the user interface elements changes status.

Description

TECHNICAL FIELD

The present invention relates generally to data processing systems and, more particularly, to a method and system for programmatically providing direct access to user interface elements of an application program.

BACKGROUND OF THE INVENTION

People with disabilities have had difficulty being integrated into the workforce. Such integration into the workforce is hampered by difficulties in working with computers. Consequently, accessibility aids have been developed to facilitate computer operation by disabled users. For example, a "screen reader utility" is a type of accessibility aid that reads information displayed on a video display to a visually impaired user, which enables the user to utilize the computer. These screen reader utilities operate by intercepting and analyzing information sent to the video driver. That is, when an application program wants to display information on the video display, it sends the information to a video driver, which is responsible for displaying the information on the video display. For example, when an application program wants to display a character, such as an ASCII character, the program sends an ASCII character code to the video driver indicating which character of the ASCII character set to display on the video display. The screen reader intercepts this ASCII character code, determines the phonetic sounds associated with the ASCII character, and audibly outputs the phonetic sounds to the visually impaired user, so that the visually impaired user can perceive the character written to the video display.

Although these screen reader utilities work well with text, screen reader utilities do not work well with graphical elements, such as the well-known graphical elements depicted in FIGS. 1A-1D. For example, FIG. 1A depicts a window with a menu bar, a title bar, a tool bar, a combo box, a header control, a tree view, a status bar, a list view, a size grip, and a scroll bar. FIG. 1B depicts a window with a progress bar, a check box-style button and status text. FIG. 1C depicts a window with an edit box and a push button-style button, and FIG. 1D depicts a window with a tab control, a list box, and a radio button-style button.

When a program wants to display a graphical element, like a button, on the video display, it sends a bitmap to the video driver containing the representation of the button. The screen reader intercepts this bitmap, and the screen reader then attempts to interpret the bitmap. To facilitate this interpretation, the screen reader maintains a database containing both the bitmaps representing the graphical elements utilized by a particular application program and an indication of what each bitmap represents. The screen reader compares the intercepted bitmap with the bitmaps in the database to determine the graphical element represented by the intercepted bitmap, and then the screen reader audibly indicates to the user what the graphical element is. In this manner, the screen reader audibly indicates to the user what information is displayed on the video display, including both text and graphical elements.

This method for analyzing a graphical element has a drawback: the screen reader utility must know in advance the exact bitmaps of the graphical elements utilized by a particular application program. Even the slightest variation between the bitmap sent to the video driver and the corresponding bitmaps in the database will prevent a match from being made, and the screen reader will be unable to identify the graphical element. In this case, the user receives an incomplete understanding of the information on the video display, thus reducing his or her effective use of the computer system. Hence, conventional screen readers typically contain the bitmaps of the graphical elements utilized by a particular application program; conventional screen readers are thus application dependent. In other words, conventional screen readers cannot convey to a user information regarding graphical elements displayed by an application program without having a priori knowledge of the exact form of its graphical elements, and the screen reader will not work on another application program without having similar a priori knowledge. As a result of these limitations, screen readers, and accessibility aids as a whole, are unreliable and do not convey sufficient information to a disabled user so that the user can understand all elements that are displayed on the video display, including both text and graphical elements. It is thus desirable to improve the way in which accessibility aids identify the elements displayed by an application program to help integrate disabled users into the work force.

Because the detailed description assumes knowledge of various well-known object-oriented techniques, an overview of such techniques is provided below. In some cases, this overview is specific to the C++ programming language.

Two common characteristics of object-oriented programming languages are support for data encapsulation and data type inheritance. Data encapsulation refers to associating functions with data, and inheritance refers to declaring a data type in terms of other data types. Thus, inheritance provides for a hierarchy of data types, where the data types defined lower in the hierarchy share the structure or behavior of the data types defined higher in the hierarchy.

In the C++ programming language, data encapsulation and inheritance are supported through the use of classes. A class is a defined type, and a class declaration describes the data members and function members of the class. The data members and function members of a class are bound together such that the function members operate on a particular instance of the data members of the class. An instance of a class is also known as an object of the class. A class thus provides a definition for a group of objects with similar properties and common behavior.

To allocate storage for an object of a particular type (class), an object is instantiated. The term "instantiating" refers to allocating memory for data members and associating the data members with the function members. Once instantiated, the function members of the particular object can be invoked to access and manipulate the data members. In this manner, the function members implement the behavior of the object, and the object provides a structure for encapsulating data and behavior into a single entity.

To support the concept of inheritance, classes may be derived from (based upon the declaration of) other classes. A derived class is a class that inherits the characteristics--data members and function members--of its base classes. A class that inherits the characteristics of another class is a derived class, and conversely, a class that does not inherit the characteristics of another class is a primary (root) class. A class whose characteristics are inherited by another class is a base class. A derived class may inherit the characteristics of several classes and have several base classes, which is known as multiple inheritance.

A class may also specify whether its function members are virtual. Declaring that a function member is virtual allows the function to be overridden by a function of the same name and type in a derived class. If a virtual function is declared without providing an implementation, then it is referred to as a pure virtual function. A pure virtual function is a virtual function declared with the pure specifier ("=0"). If a class specifies a pure virtual function, then any derived class needs to specify an implementation for that function member before an object of that class can be instantiated. A class which contains at least one pure virtual function member is an abstract class.

FIG. 2 is a block diagram illustrating typical data structures used to represent an object. An object comprises instance data (data members) and function members, which implement the behavior of an object. The data structures used to represent an object comprise instance data structure 201, virtual function table 202, and the function members 203, 204, 205. The instance data structure 201 contains a pointer to the virtual function table 202 and contains data members. The virtual table 202 contains an entry for each virtual function member defined for the object. Each entry contains a reference to the code that implements the corresponding function member. In the following, an object will be described as an instance of a class as defined by the C++ programming language. One skilled in the art, however, will appreciate that other object models can be defined using other programming languages.

An advantage of using object-oriented techniques is, these techniques can be used to facilitate the sharing of objects. For example, a program implementing the function members of an instantiated object (a "server program") can share the object with another program (a "client program"). To allow an object of an arbitrary class to be shared with a client program, interfaces are defined through which an object can be accessed without the need for the client program to have access to the class implementation at compile time. An interface is a named set of logically related function members ("methods") and data members ("properties"). In C++, an interface is implemented as an abstract class with no data members and whose virtual functions are all pure. Thus, an interface provides a published protocol for two programs to communicate. Interfaces are typically used for derivation: a program defines (implements) classes that provide implementations for the interfaces the classes are derived from. Thereafter, objects are created as instances of these derived classes. Objects instantiated from a derived class implementing particular interfaces are said to "support" the interfaces. Typically, an object supports one or more interfaces depending upon the desired functionality.

The above-described object-oriented techniques have been successfully employed in Microsoft Corporation's Object Linking and Embedding (OLE) system. OLE is an object-based system in which each object conforms to the component object model by exposing a number of predefined interfaces. One of these interfaces is the IUnknown interface which contains the QueryInterface function member, and this function member provides access to all of the other interfaces supported by the object. Another interface supported by objects conforming to the component object model is the IDispatch interface, which allows the function members of an object to be invoked by a client without the client knowing at compile time the definition of the function member, which is known as dynamic binding. The IDispatch interface is described in greater detail in U.S. Pat. No. 5,515,536, entitled "Method and System for Invoking Methods of an Object Through a Dispatching Interface," issued May 7, 1996, which is hereby incorporated by reference. More generally, however, both OLE and the component object model are described in greater detail in Brockschmidt, Inside OLE, Second Edition, Microsoft Press (1995), at pages 1-143 and 635-730.

SUMMARY OF THE INVENTION

An architecture is provided that enables an accessibility aid to directly access and manipulate user interface elements of an application program programmatically. Moreover, such access and manipulation occurs in an application-independent manner, so that an accessibility aid utilizing the architecture can access any application program that conforms to the architecture, without the accessibility aid needing any prior knowledge of the application program or its user interface elements. A "user interface element" is information displayed on the video display that conveys meaning to a user, including both text and graphical elements like buttons, scroll bars, toolbars, edit boxes, and windows. User interface elements typically have both a visual representation displayed on the video display and an implementation, which is the code and data implementing the user interface element. The architecture provides an accessibility aid with direct access to the implementation of user interface elements, thus enabling the accessibility aid to both examine various characteristics of the user interface element and manipulate these characteristics, which may affect its visual representation.

By utilizing the architecture, accessibility aids can access and manipulate the user interface elements of any application program without the need to have a priori knowledge of the application program or its user interface. An accessibility aid communicates directly to the application program and to the implementation of its user interface elements via the architecture, and the accessibility aid can thus obtain a complete and rich description of the application program's user interface elements and can manipulate the user interface elements. Upon receiving the description of the user interface elements, the accessibility aid can convey this description to a disabled user via means which they can perceive, such as through audible output.

In accordance with a first aspect of the present invention, a method is practiced in a computer system for providing direct access to user interface elements of an application program programmatically. The application program exposes a programmatic access component for use by a client. The application program further receives a programmatic request identifying a user interface element to access via the programmatic access component and directly accesses the identified user interface element on behalf of the client responsive to the received programmatic request.

In accordance with a second aspect of the present invention, a method is practiced in a computer system for accessing user interface elements of an application program by a client. The client requests a reference to an interface supported by the application program, where the interface contains at least one function member that provides access to at least one of the user interface elements. Furthermore, the client receives the reference to the interface and invokes the function member to directly access and manipulate the user interface element in an application independent manner.

In accordance with a third aspect of the present invention, a method is practiced in a computer system for accessing a user interface element of a computer program. The user interface element has a class definition that is able to be instantiated into an object. The method starts the computer program, where the class definition is not instantiated, and the method receives a request to access the user interface element. In addition, the method instantiates the class definition such that an object is created that represents the user interface elements responsive to receiving the request.

In accordance with a fourth aspect of the present invention, a method is practiced in a computer system having an application program with a user interface having user interface elements and an operating system that manages the application program. The application program determines when a state of one of the user interface elements has changed, and the application program generates an event to notify listeners that the state of the user interface element has changed.

In accordance with a fifth aspect of the present invention, a method is practiced in a computer system for accessing user interface elements of an application program. The user interface elements are related to each other by a plurality of relations. The application program receives a programmatic request from a client indicating a user interface element and indicates one of the plurality of relations. Furthermore, the application program returns to the client a reference to a related user interface element that is related to the identified user interface element via the indicated relation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A-1D depict well-known graphical elements that are displayed as part of a user interface of a computer program.

FIG. 2 is a block diagram illustrating conventional data structures used to represent an object.

FIG. 3 depicts a client accessing an IAccessible interface of a server in accordance with a preferred embodiment of the present invention.

FIG. 4 depicts the interactions between a client, an operating system, and a server to obtain a pointer to the IAccessible interface of the server.

FIG. 5 depicts a computer system that is suitable for practicing a preferred embodiment of the present invention.

FIGS. 6A and 6B depict a flowchart of the steps performed by the accNavigate function member on the IAccessible interface.

FIG. 7 depicts a flowchart of the steps performed by the server in response to receiving a WM_GetObject message.

FIG. 8 depicts a flowchart of the steps performed by the LResultFromObject function.

FIG. 9 depicts a flowchart of the steps performed by the ObjectFromLResult function.

FIG. 10 depicts an event table utilized by an access component of the operating system.

FIG. 11 depicts a flowchart of the steps performed by the NotifyWinEvent function provided by the access component of the operating system.

FIG. 12 depicts a flowchart of the steps performed by the AccessibleObjectFromPoint function provided by the access component of the operating system.

FIG. 13 depicts a more detailed diagram of the interactions between the client, the access component, and the server depicted in FIG. 3.

FIG. 14 depicts a flowchart of the steps performed by a client when determining the menu structure of the server.

DETAILED DESCRIPTION OF THE INVENTION

A preferred embodiment of the present invention provides an architecture that enables an accessibility aid to directly access and manipulate user interface elements of an application program programmatically. Moreover, such access and manipulation occurs in an application-independent manner, so that an accessibility aid utilizing the preferred architecture can access any application program that conforms to the preferred architecture, without the accessibility aid needing any prior knowledge of the application program or its user interface elements. A "user interface element" is information displayed on the video display that conveys meaning to a user, including both text and graphical elements like buttons, scroll bars, toolbars, edit boxes, and windows. User interface elements typically have both a visual representation displayed on the video display and an implementation, which is the code and data implementing the user interface element. The preferred architecture provides an accessibility aid with direct access to the implementation of user interface elements, thus enabling the accessibility aid to both examine various characteristics of the user interface element and manipulate these characteristics, which may affect its visual representation.

By utilizing the preferred architecture, accessibility aids can access and manipulate the user interface elements of any application program without the need to have a priori knowledge of the application program or its user interface. An accessibility aid communicates directly to the application program and to the implementation of its user interface elements via the preferred architecture, and the accessibility aid can thus obtain a complete and rich description of the application program's user interface elements and can manipulate the user interface elements. Upon receiving the description of the user interface elements, the accessibility aid can convey this description to a disabled user via means which they can perceive, such as through audible output.

An example of an accessibility aid utilizing the preferred architecture is a screen reader utility. The screen reader utility receives audible commands from the user and invokes the commands on an application program utilizing the preferred architecture. These commands both access and manipulate the user interface elements of the application program.

The preferred architecture provides accessibility aids (or clients) with functionality that is unable to be performed by conventional systems. This functionality is useful in conveying a more rich, complete, and accurate description of the user interface elements of an application program. For example, clients can receive user interface element-based events that are generated by and originate from an application program, so that the clients are notified when a user interface element changes status. Such events include indications of when a button can no longer be activated or an indication in a spreadsheet program of which spreadsheet cell currently has focus. The term "focus" refers to an indication of a user interface element, such as a spreadsheet cell or window, to which all input from the keyboard is currently directed. In conventional systems, clients cannot receive such events originating from an application program, and the user is consequently unable to completely and accurately understand the user interface of the application program as it changes, which significantly reduces a user's ability to interact with the application program.

The preferred architecture also provides clients with the ability to navigate between user interface elements. User interface elements are typically displayed such that each user interface element has a relationship to the other user interface elements. Such relationships include each user interface element being part of a sequence. In some situations, the application program knows the sequence, and this sequence is not apparent to even sighted users until they manipulate the user interface. For example, a number of edit boxes may be related in a sequence such that when the user depresses the tab key, the cursor moves to the next edit box in the sequence. Such a relationship, however, has previously been imperceptible to a user who cannot view the visual representation of the user interface. Additionally, many user interface elements have a parent/child relationship. Two user interface elements have a parent/child relationship when the child is displayed within the borders of the parent. For example, a toolbar with buttons that perform particular actions has a parent relationship to the child buttons. Again, such a relationship has previously been imperceptible to a user who cannot view the visual representation of the user interface.

Through the use of the preferred architecture, clients can navigate between the user interface elements, determine their relationship, and convey this relationship to users who cannot visually perceive such relationships. The preferred architecture allows a client to navigate through a sequence of user interface elements to obtain the first, last, previous, or next user interface element in the sequence, and the preferred architecture allows a client to navigate through user interface elements in a parent/child relationship by identifying all children of a parent or identifying a parent of a child. Additionally, the preferred architecture enables a client to navigate through user interface elements using a direction relative to the video display, such as up, down, left or right. Such directional navigation allows a user to identify relationships among user interface elements, like text, on the video display. For example, a user can determine that data displayed on the computer display is displayed as a table with two columns, which conveys additional meaning to the user and which further enhances their effective use of both the computer and the application program.

Furthermore, using the preferred architecture, accessibility aids can query an application program for a textual description of a user interface element. For example, if the application program displays a scissor button that cuts selected text from a document, the textual description may be "this is a button that displays a pair of scissors." Using conventional systems, however, unless the accessibility aid has a priori knowledge, this information is unavailable to a user who cannot perceive the visual representation of the user interface, and this unavailability restricts the user's ability to interact with the computer system.

The preferred architecture also enables a user to determine what user interface element is located at a particular location on the video display. Typically, conventional accessibility aids, like screen readers, operate relative to a reference point, similar to a cursor, by reading the word under the reference point. However, if the reference point is located over a graphical element which the conventional screen reader cannot understand, no information is conveyed to the user, and the user has no way of knowing what is under the reference point, thus receiving an incomplete understanding of the user interface. Using the preferred architecture, the user can invoke a client to retrieve from the application program an indication of the user interface element at a particular location, which enables a user to move the reference point over the display and identify the user interface element at any given location to receive a complete description of the user interface.

Another advantage of the preferred architecture is that it allows a client to determine the menu structure of an application program without the application program having to display its user interface. Typically, in conventional systems, the only way that an application program's menu structure can be determined is through an accessibility aid "reading" the visual representation of the menu structure. Using the preferred architecture, however, an accessibility aid can query an application program to determine its menu structure, including all menus, submenus, and items on a submenu. In this manner, a user who is unable to view the visual representation of the menu structure can determine all of the commands supported by a particular application program, without the program having to display the menu structure, which saves processing time. In addition to the advantages described above, other benefits of using the preferred architecture will be apparent to those skilled in the art upon reading the following description.

OVERVIEW AND EXAMPLE

As part of the preferred architecture, an application program supports the IAccessible interface, which allows clients to reap the benefits of using the preferred architecture. As shown in FIG. 3, a client 302, such as an accessibility aid, accesses a server 304, such as an application program with user interface elements, via the IAccessible interface 306. The IAccessible interface 306 contains function members which allow the client 302 to directly access and manipulate the implementation of the user interface elements of the server 304. These function members allow the client 302 to access the user interface elements in the following ways: by navigating through the user interface elements, by retrieving the name for each user interface element, by determining whether each user interface element is visible or hidden, by retrieving the text description for each user interface element, by identifying the location of the user interface element, by determining the parent or child of a user interface element, and by determining the value of the user interface element. Some user interface elements have an associated value. For example, a clock user interface element has a value of the current time.

Additional function members on the IAccessible interface, instead of accessing the user interface element, enable the client to manipulate or modify characteristics of the user interface element. For example, these function members can perform the default action associated with the user interface element, select the user interface element, or set the value of the user interface element. Each user interface element capable of performing an action, such as a button, typically has an associated default action, like being pressed. One of the function members on the IAccessible interface allows a client to invoke the default action of a user interface element. Also, some user interface elements, such as text, are capable of being selected, which is typically indicated by highlighting the user interface element (e.g., the text) on the video display, and one of the function members on the IAccessible interface allows a client to select a user interface element. When a user interface element is selected, it can be manipulated in many well-known ways, like cutting, pasting, moving, or dragging and dropping. Although only a subset of the function members have been briefly described, each of the function members are described in greater detail below.

The server 304 typically supports the IAccessible interface 306 for each of its user interface elements. In some situations, however, the server 304 may only support the IAccessible interface 306 for a parent user interface element having a number of children. In this situation, the server 304 does not incur the overhead of creating an IAccessible interface for each of the children and thus saves a significant amount of memory.

The IAccessible interface can be accessed by the client in a number of ways. First, the client can invoke the QueryInterface function member on the IUnknown interface to obtain a reference to the IAccessible interface. The QueryInterface function member is supported by all interfaces of objects conforming to the component object model of OLE. Second, the client may invoke functions of the operating system to obtain a reference to an IAccessible interface for a user interface element. This second method of accessing the IAccessible interface is perhaps the most beneficial, because the client may not know exactly in which of the server's user interface elements it is interested, and these functions help determine which user interface element's IAccessible interface to obtain based on various characteristics of the user interface element. For example, the client may only have a location relative to the video display. In this situation, the client does not know which user interface element to query and therefore invokes a function of the operating system to obtain a reference to the IAccessible interface for that user interface element given its location. It is therefore advantageous to be able to query the operating system to retrieve a reference to the IAccessible interface for a user interface element.

To provide the client with the benefits of using the preferred architecture, both the server and the operating system perform various functionality. To help describe this functionality, FIG. 4 depicts an example of the interactions between the client 302, the operating system 402, and the server 304. The client 302 is typically an accessibility aid, the server 304 is an application program with user interface elements, and the operating system 402 performs functionality that facilitates the interactions between the client and the server. As stated above, the client 302 can receive events from the server 304 and perform processing in response to these events. For example, the server 304 may display a "scissor button" that, upon depression, cuts selected text from a document. The scissor button, however, is only invokable when text is selected; when no text is selected, the button is inactive and takes on a gray appearance. The client 302 may want to be notified when the scissor button becomes activated, and after being notified, the client may then want to obtain the IAccessible interface for the scissor button, so that the client may activate the scissor button.

FIG. 4 depicts an overview of the interactions between the client 302, the server 304, and the operating system 402 to accomplish the functionality of the scissor button example. First, when the client 302 wants to be notified when the scissor button becomes activated, the client registers with the operating system 402. Second, sometime later, the server 304 generates an event notification indicating that the scissor button has become activated. Third, the operating system 402 receives the event notification and passes the event notification to the client 302. Fourth, upon receiving the event notification, the client 302 requests a reference to the IAccessible interface from the operating system 402 so that the client may activate the scissor button. Fifth, the operating system 402 passes this request to the server 304, and sixth, the server replies to the request with a reference to the IAccessible interface for the user interface element that generated the event. Seventh, the operating system 402 then returns the interface pointer to the client 302. Thereafter, the client 302 can activate the scissor button and, more generally, invoke the function members on the IAccessible interface to both access and manipulate the scissor button.

Implementation Details

FIG. 5 depicts a computer system 500 that is suitable for practicing a preferred embodiment of the present invention. The computer system 500 contains a memory 502, a secondary storage device 504, a central processing unit (CPU) 506, an input device 508, and a video display 510. The memory 502 contains the client 302, the server 304, and the operating system 402. The operating system 402 contains an access component 512 that facilitates communication between the client 302 and the server 304 for accessibility-related processing. An example of an operating system 402 suitable for use in the computer system 500 is the Windows.RTM. 95 Operating System available from Microsoft Corporation of Redmond, Washington. The client 302 utilizes the function members on the IAccessible interface 306 exposed by the server 304. The server 304, as stated above, exposes the IAccessible interface 306 for a number of its user interface elements and also exposes an IUnknown interface, like all programs conforming to the OLE architecture. Although a preferred embodiment of the present invention is described below as conforming to the OLE architecture, one skilled in the art will appreciate that the present invention can work in other object-based systems or nonobject-based systems.

In order to conform to the preferred architecture, the server and the access component support various aspects of the architecture that are described below. In the following table, each component is indicated with the various aspects of the referred architecture that it supports.

        Component          Aspect of preferred architecture supported
        Server             IAccessible Interface
                           Handling a WM_GetObject message
        Access Component   LResultFromObject function
                           ObjectFromLResult function
                           SetWinEventHook function
                           NotifyWinEvent function
                           AccessibleObjectFromPoint function
                           AccessibleObjectFromEvent function
                           AccessibleObjectFromWindow function

    State                              Meaning
    STATE_SYSTEM_ALERT_HIGH            It is important that this information be
     conveyed
                                       to the user immediately. For example, a
     battery
                                       level indicator reaching a critical
     level conveys
                                       truly urgent information, so a screen
     reader utility
                                       should announce this information
     immediately.
    STATE_SYSTEM_ALERT_LOW             This information is of low priority, so
     the user
                                       need not be immediately informed that it
                                       occurred.
    STATE_SYSTEM_ALERT_MEDIUM          The user should be informed that this
     information
                                       is available, but the informational
     content need
                                       not be conveyed immediately. For
     example, when
                                       a battery level indicator reaches a low
     level, it
                                       should generate a medium-level alert.
     Screen
                                       reader utilities could then generate a
     sound to let
                                       the user know that important information
     is
                                       available, without actually interrupting
     the user's
                                       work. The user could then query the
     alert
                                       information at his or her leisure.
    STATE_SYSTEM_ANIMATED              This status indicates that the user
     interface
                                       element's appearance is changing rapidly
     or
                                       constantly.
    STATE_SYSTEM_BUSY                  This status indicates that the user
     interface
                                       element cannot accept input at this
     time.
    STATE_SYSTEM_CHECKED               This status indicates that the user
     interface
                                       element's check box is selected.
    STATE_SYSTEM_FOCUSABLE             This status indicates that the user
     interface
                                       element can accept the keyboard input
     focus.
    STATE_SYSTEM_FOCUSED               This status indicates that the user
     interface
                                       element has the keyboard input focus.
    STATE_SYSTEM_INVISIBLE             This status indicates that the user
     interface
                                       element is hidden or invisible.
    STATE_SYSTEM_MARQUEED              This status indicates that text is being
     scrolled or
                                       moved.
    STATE_SYSTEM_MIXED                 This status indicates a three-state
     check box or
                                       toolbar button.
    STATE_SYSTEM_MULTISELECTA          This status indicates that the user
     interface
    BLE                                element can select multiple items.
    STATE_SYSTEM_OFFSCREEN             This status indicates that the user
     interface
                                       element is currently off the display and
     not
                                       visible.
    STATE_SYSTEM_PRESSED               This status indicates that the user
     interface
                                       element is pressed.
    STATE_SYSTEM_READONLY              This status indicates that the user
     interface
                                       element is read only.
    STATE_SYSTEM_SELECTABLE            This status indicates that the user
     interface
                                       element can accept selection.
    STATE_SYSTEM_SELECTED              This status indicates that the user
     interface
                                       element is currently selected.

    Term        Description
    Shortcut    Key combinations that invoke an action. For example,
    keys        CTRL +0 is often used to invoke the Open file
                common dialog box.
    Access keys Single keys, usually shown underlined in a drop-down
                menu or dialog box, that invoke an action when pressed.
                For example, if a user activates an application's File menu,
                the "o" key often invokes the Open file common dialog
                box. Access keys are usually only available when the user
                interface element, like a menu, has keyboard focus. For
                example, access keys are available for items on a menu
                when any part of that menu has focus.

    NAVDIR_UP              Requesting user interface elements
                           physically above the current one
                           relative to the video display.
    NAVDIR_DOWN            Requesting user interface elements
                           physically below the current one.
    NAVDIR_LASTCHILD       Go to the last child of this user
                           interface element.
    NAVDIR_FIRSTCHILD      Go to the first child of this user
                           interface element.
    NAVDIR_LEFT            Requesting user interface elements
                           physically to the left of the current one.
    NAVDIR_RIGHT           Requesting user interface elements
                           physically to the right of the current one
                           relative to the video display.
    NAVDIR_NEXT            The next logical location or user
                           interface element, generally a
                           "sibling" to the current user interface
                           element. For example, in a dialog box,
                           the TAB key moves to the next logical
                           location, although this can be
                           represented in any number of different
                           physical directions.
    NAVDIR_PREVIOUS        The previous logical location or user
                           interface element. In a dialog box, the
                           SHIFT+TAB key combination moves
                           to the previous logical control, although
                           this may be in any number of physical
                           directions visually on the screen. For
                           example, in vertical toolbars, logically
                           the previous button is often the button
                           physically above (NAVDIR_UP) the
                           current one, whereas in horizontal
                           toolbars, logically the previous button
                           is generally the button physically to the
                           left (NAVDIR_LEFT) of the
                           current one.

        VT_DISPATCH        One child was selected and the address
                           of its IDispatch interface is contained
                           in the data.
        VT_EMPTY           No children were selected.
        VT_I4              One child was selected.
        VT_UNKNOWN         Multiple children were selected.

    Event                                   Description
    EVENT_SYSTEM_SOUND                      The system generates this event
     when a system
                                            sound (e.g., for menus) is played.
     The system will
                                            generate this event even if no
     sound is audible
                                            (for example, lack of a sound file
     to play, no
                                            sound card, etc.).
    EVENT_SYSTEM_ALERT                      An alert was generated. Message
     boxes generate
                                            this event when they are displayed.
    EVENT_SYSTEM_FOREGROUND                 The foreground window (the window
     appearing
                                            closest to the user) is changing.
    EVENT_SYSTEM_MENUPOPUPSTART             A pop-up menu is being displayed.
    EVENT_SYSTEM_MENUPOPUPEND               A pop-up menu is being removed.
    EVENT_SYSTEM_CAPTURESTART               A window received mouse capture
     such that the
                                            mouse cursor is traversing over the
     window.
    EVENT_SYSTEM_CAPTUREEND                 A window lost mouse capture.
    EVENT_SYSTEM_DRAGDROPSTART              An application program is about to
     enter drag-
                                            and-drop mode where a drag-and-drop
     operation
                                            is performed.
    EVENT_SYSTEM_DRAGDROPEND                An application program is about to
     exit drag-and-
                                            drop mode where a drag-and-drop
     operation is
                                            performed.
    EVENT_SYSTEM_DIALOGSTART                A dialog box was displayed.
    EVENT_SYSTEM_DIALOGEND                  A dialog box was removed.
    EVENT_SYSTEM_SCROLLINGSTART             A scroll bar is being manipulated.
    EVENT_SYSTEM_SCROLLINGEND               A scroll bar is no longer being
     manipulated.
    EVENT_SYSTEM_SWITCHSTART                The user pressed ALT + TAB to move
     to another
                                            program running in the computer.
    EVENT_SYSTEM_SWITCHEND                  The user released ALT + TAB.
    EVENT_SYSTEM_MINIMIZESTART              A window is about to be minimized
     or
                                            maximized.
    EVENT_SYSTEM_MINIMIZEEND                A window was minimized or
     maximized.
    EVENT_OBJECT_CREATE                     Indicates that a window was
     created. In the case
                                            of windows only, both the parent
     and the child
                                            send this event. The child window
     sends the event
                                            when created, and the parent sends
     it after
                                            creation is complete.
    EVENT_OBJECT_DESTROY                    Indicates that a window was
     destroyed. In the
                                            case of window objects only, both
     the parent and
                                            the child send this event. The
     child window sends
                                            the event when it is destroyed, and
     the parent
                                            sends it after destruction is
     complete.
    EVENT_OBJECT_SHOW                       A user interface element is being
     displayed.
    EVENT_OBJECT_HIDE                       A user interface element is being
     hidden.
    EVENT_OBJECT_REORDER                    A user interface element's children
     are changing
                                            their z-order placements. The
     "z-order" refers to
                                            the depthwise appearance of the
     windows on the
                                            video display.
    EVENT_OBJECT_FOCUS                      A user interface element is
     receiving keyboard
                                            focus.
    EVENT_OBJECT_SELECTION                  A user interface element selection
     change
                                            occurred.
    EVENT_OBJECT_SELECTIONADD               A user interface element was added
     to the
                                            selection.
    EVENT_OBJECT_SELECTIONREMOVE            A user interface element. was
     removed from the
                                            selection.
    EVENT_OBJECT_STATECHANGE                A user interface element's state
     changed. For
                                            example, a state change can occur
     when a button
                                            has been pressed or released, or
     when a button is
                                            being enabled or disabled.
    EVENT_OBJECT_LOCATIONCHANGE             A user interface element is
     changing its location,
                                            shape, or size.
    EVENT_OBJECT_NAMECHANGE                 A user interface element's Name
     property
                                            changed.
    EVENT_OBJECT_DESCRIPTIONCHANGE          A user interface element's
     Description property
                                            changed.
    EVENT_OBJECT_VALUECHANGE                A user interface element's Value
     property
                                            changed.
    EVENT_OBJECT_PARENTCHANGE               A user interface element has a new
     parent object.
    EVENT_OBJECT_HELPCHANGE                 A user interface element's Help
     property changed.
    EVENT_OBJECT_DEFACTIONCHANGE            A user interface element's
     DefaultAction property
                                            changed.
    EVENT_OBJECT_ACCELERATORCHANGE          A user interface element's
     KeyboardShortcut
                                            property changed.

• Inventors
  Oppermann, Charles; Brown, Nathaniel S.; Lowney, Gregory C.; Atkinson, Robert G.; Butler, Laura J.; Wong, Peter Kam-Ho;
• Assignee
  Microsoft Corporation (Redmond, WA)
• Published
  Dec-25-2001
• Current US Classes:
  719/310
  719/328
• Application #
  815100
• International Classes
  G06F 009/00
• Field of Search
  709/300 709/303 709/310 709/313 709/328 709/329 395/326 395/333 345/338 345/327 345/158 345/352
• Examiner
  Maung; Zarni
• Agent
  Banner & Witcoff, Ltd.
• US Patent References:
  5363121
  5394521
  5533183
  5717877
  5748927
  5754176
  5801701
  5901313
  6057831