Systems and methods for implementing modular DOM (Document Object Model)-based multi-modal browsers

Abstract

Systems and methods for building multi-modal browsers applications and, in particular, to systems and methods for building modular multi-modal browsers using a DOM (Document Object Model) and MVC (Model-View-Controller) framework that enables a user to interact in parallel with the same information via a multiplicity of channels, devices, and/or user interfaces, while presenting a unified, synchronized view of such information across the various channels, devices and/or user interfaces supported by the multi-modal browser. The use of a DOM framework (or specifications similar to DOM) allows existing browsers to be extended without modification of the underling browser code. A multi-modal browser framework is modular and flexible to allow various fat client and thin (distributed) client approaches.

Claims

What is claimed is:

1. A multi-modal browser, comprising:

a plurality of modality-specific browsers, and

a multi-modal shell that parses and processes a modality-independent representation of an application to generate modality-specific representations of the application which are rendered by corresponding modality-specific browsers to generate modality-specific views of the application, and to manage synchronization of I/O (input/output) events across each modality-specific view,

wherein each modality-specific browser comprises:

an API (application programming interface) that enables the multi-model shell to control the browser and access events; and

a wrapper interface that support synchronization between the modality-specific views by event filtering and exhchanging synchronization information between the multi-modal shell and the browser,

wherein the multi-modal shell comprises:

a model manager for maintaining a dialog state of the application;

a TAV (transformation/adaption/view preparation) manager for preparing and transforming pages or page snippets; and

a synchronization manager for managing event notifications to the browsers.

2. The multi-modal browser of claim 1, wherein the API for a modality-specific browser comprises a DOM (document object model) interface.

3. The multi-modal browser of claim 2, wherein the wrapper interface comprises methods for DOM event filtering.

4. The multi-modal browser of claim 1, wherein the multi-modal shell maintains and updates a dialog state of the application.

5. The multi-modal browser of claim 1, wherein the multi-modal browser comprises a fat client framework.

6. The multi-modal browser of claim 1, wherein the multi-modal browser comprises a distributed framework.

7. The multi-modal shell of claim 1, wherein the multi-modal shell comprises a distributed framework.

8. The multi-modal browser of claim 1, wherein the plurality of modality-specific browsers comprise a WML (wireless markup language) browser and a VoiceXML browser.

9. The multi-modal browser of claim 8, further comprising an audio system for capturing and encoding speech data, and a plurality of speech engines for processing speech data.

10. A distributed WAP (wireless application protocol) multi-modal browser system, comprising:

a GUI (graphical user interface) browser that resides on a client device, wherein the GUI browser comprises a DOM (document object model) interface for controlling the GUI browser and managing DOM event notifications, and a wrapper interface for filtering events;

a speech application server comprising:

a voice browser, wherein the voice browser comprises a DOM interface for controlling the voice browser and managing DOM event notifications, and a wrapper interface for filtering events; and

an audio system for capturing and encoding speech data; and one or more

speech engines for processing speech data,

wherein the speech application server is distributed over a network;

a multi-modal shell system for parsing and processing a modality-independent application and managing synchronization of I/O (input/output) events between the GUI and voice browsers; and

a communication manager that employs protocols for transporting encoded voice data to the speech engines and protocols for enabling synchronization of the GUI browser.

11. The WAP multi-modal browser of claim 10, wherein the GUI browser comprises a WML (wireless markup language) browser and wherein the voice browser comprises a VoiceXML browser.

12. The WAP multi-modal browser of claim 10, wherein the speech application server supports protocols for enabling remote control of the engines for server side speech processing.

13. The WAP multi-modal browser of claim 10, wherein the multi-modal shell system is distributed over the network.

14. A multi-modal browser, comprising:

a plurality of modality-dependent browsers, and

a multi-modal shell for parsing and processing a modality-independent application and managing synchronization of I/O (input/output) events across each view generated by the plurality of modality-dependent browsers,

wherein each modality-dependent browser comprises:

an API (application programming interface) for controlling the browser and for managing events; and

an external wrapper interface comprising synchronization protocols for supporting synchronization of the browser, and

wherein the multi-modal shell comprises:

a model manager for maintaining a dialog state of the application;

a TAV (transformation/adaption/view preparation) manager for preparing and transforming pages or page snippets; and

a synchronization manager for managing event notifications to the browsers.

Description

BACKGROUND

The present invention relates generally to systems and methods for building multi-modal browsers applications and, in particular, to systems and methods for building modular multi-modal browsers using a DOM (Document Object Model) and MVC (Model-View-Controller) framework that enables a user to interact in parallel with the same information via a multiplicity of channels, devices, and/or user interfaces, while presenting a unified, synchronized view of such information across the various channels, devices and/or user interfaces supported by the multi-modal browser.

The computing world is evolving towards an era where billions of interconnected pervasive clients will communicate with powerful information servers. Indeed, this millennium will be characterized by the availability of multiple information devices that make ubiquitous information access an accepted fact of life. This evolution towards billions of pervasive devices being interconnected via the Internet, wireless networks or spontaneous networks (such as Bluetooth and Jini) will revolutionize the principles underlying man-machine interaction. In the near future, personal information devices will offer ubiquitous access, bringing with them the ability to create, manipulate and exchange any information anywhere and anytime using interaction modalities most suited to the an individual's current needs and abilities. Such devices will include familiar access devices such as conventional telephones, cell phones, smart phones, pocket organizers, PDAs and PCs, which vary widely in the interface peripherals they use to communicate with the user.

The increasing availability of information, along with the rise in the computational power available to each user to manipulate this information, brings with it a concomitant need to increase the bandwidth of man-machine communication. The ability to access information via a multiplicity of appliances, each designed to suit the individual's specific needs and abilities at any given time, necessarily means that these interactions should exploit all available input and output (I/O) modalities to maximize the bandwidth of man-machine communication. Indeed, users will come to demand such multi-modal interaction in order to maximize their interaction with information devices in hands-free, eyes-free environments.

The current networking infrastructure is not configured for providing seamless, multi-modal access to information. Indeed, although a plethora of information can be accessed from servers over a communications network using an access device (e.g., personal information and corporate information available on private networks and public information accessible via a global computer network such as the Internet), the availability of such information may be limited by the modality of the client/access device or the platform-specific software applications with which the user is interacting to obtain such information.

By way of example, one of the most widely used methods for accessing information over a communications network is using a conventional HTML browser to access information over the WWW (world wide web) using, for example, portals such as Yahoo! and AOL. These portals typically include a directory of Web sites, a search engine, news, weather information, e-mail, stock quotes, etc. Typically, only a client/access device having full GUI capability can take advantage of such Web portals for accessing information.

Other conventional portals and access channels include wireless portals/channels that are typically offered by telephone companies or wireless carriers (which provide proprietary content to subscribing users and/or access to the Internet or a wireless portion of the Internet, with no restrictions or access control). These wireless portals may be accessed via WAP (wireless application protocol) by client/access devices (via a WAP browser) having limited GUI capabilities declaratively driven by languages such as WML (wireless markup language), CHTML (compact hypertext markup language) such as NTT DocoMo imode), or XHTML (eXtensible HTML) Mobile Profile (XHTML-MP) as specified by WAP 2.0. WAP together with WML and XHTML-MP and iMode with CHRML allow a user to access the Internet over a cellular phone with constrained screen rendering and limited bandwidth connection capabilities. Currently, wireless portals do not offer seamless multi-modal access (such as voice and GUI) or multi-device (more than one device simultaneously available) regardless of the access device. Instead, a separate voice mode is used for human communication and a separate mode is used for WAP access and WML browsing.

In addition, IVR services and telephone companies can provide voice portals having only speech I/O capabilities. The IVR systems may be programmed using, e.g., proprietary interfaces (state tables, scripts beans, etc.) or VoiceXML (a current speech ML standard) and objects. With a voice portal, a user may access an IVR service and perform voice browsing using a speech browser (or using telephone key pads). Unfortunately, a client/access device having only GUI capability would not be able to directly access information from a voice portal. Likewise, a client/access device having only speech I/O would not be able to access information in a GUI modality.

Currently, new content and applications are being developed for Web accessibility with the intent of delivering such content and application via various channels with different characteristics, wherein the content and applications must be adapted to each channel/device/modality. These "multi-channel applications" (an application that provides ubiquitous access through different channels (e.g., VoiceXML, HTML), one channel at a time) do not provide synchronization or coordination across views of the different channels.

One challenge of multi-channel applications/content is that since new devices and content emerge continuously, this adaptation must be made to work for new devices not originally envisioned during the development process. In addition, it is important to be able to adapt existing content that may not have been created with this multi-channel or multi-modal deployment model in mind.

Further challenges of multi-channel applications is that, notwithstanding that multi-channel applications enable access to information through any device, it is difficult to enter and access data using small devices since keypads and screens are tiny. Further, voice access is more prone to errors and voice output is inherently sequential. One interaction mode does not suit all circumstances: each mode has its pros and cons. One optimal interaction mode at a moment can no more be optimal at another moment or for another user. All-in-one devices are no panacea, and many different devices will coexist. In fact, no immediate relief is in sight for making multi-channel e-business easier. Devices are getting smaller, not larger. Devices and applications are becoming more complex requiring more complex or efficient user interfaces. Adding color, animation, streaming, etc. does not simplify the e-business issues mentioned above. Considering these factors leads to the conclusion that an improved user interface will accelerate the growth of mobile e-business.

Accordingly, systems and methods for building and implementing user interfaces an applications that operate across various channels and information appliances, and which allow a user to interact in parallel with the same information via a multiplicity of channels and user interfaces, while presenting a unified, synchronized view of information across the various channels, are highly desirable. Indeed, there will be an increasingly strong demand for devices and browsers that provide such capabilities.

SUMMARY OF THE INVENTION

The present invention relates generally to systems and methods for building multi-modal browser applications and, in particular, to systems and methods for building modular multi-modal browsers based on a DOM (Document Object Model) and MVC (Model-View-Controller) framework that enables a user to interact in parallel with the same information via a multiplicity of channels, devices, and/or user interfaces, while presenting a unified, synchronized view of such information across the various channels, devices and/or user interfaces supported by the multi-modal browser.

In one aspect of the present invention, a multi-modal browser is based on a MVC (Model-View-Controller) framework, wherein a single information source, Model M (comprising a modality-independent representation of an application) is mapped to a plurality of Views (e.g., different synchronized channels) and manipulated via a plurality of Controllers C1, C2 and C3 (e.g., different browsers such as a speech browser, a GUI browser and a multi-modal browser or different devices). The Controllers act on, transform and manipulate the same underlying Model M to provide synchronized Views. The synchronization of the Views is achieved by generating all Views from, e.g., a single unified representation that is continuously updated.

In one aspect of the invention, a MVC multi-modal browser comprises:

a model manager for managing a model comprising a modality-independent representation of an application, and

a plurality of channel-specific controllers, wherein each controller processes and transforms the model to generate a corresponding channel-specific view of the model, wherein the channel-specific views are synchronized by the model manager such that a user interaction in one channel-specific view is reflected in another channel-specific view.

In another aspect, the model manager updates and maintains a dialog state of the model, wherein a user interaction within a given view will update a dialog state of the model. Further, the model manager maintains conversation history and context.

In yet another aspect, the multi-modal shell of the multi-modal browser supports an application model based on a single authoring framework and a multiple authoring framework.

In another aspect of the invention, a multi-modal browser comprises:

a plurality of modality-dependent browsers; and

a multi-modal shell for parsing and processing a modality-independent application and managing synchronization of I/O (input/output) events across each view generated by the plurality of modality-dependent browsers, wherein each modality-dependent browser comprises:

an API (application programming interface) for controlling the browser and for managing events; and

a wrapper interface comprising synchronization protocols for supporting synchronization of the browser.

Preferably, the associated API for a modality-dependent browser comprises a DOM (document object model) interface and the associated wrapper interface comprises methods for DOM event filtering.

In yet another aspect of the invention, the multi-modal browser is modular allowing fat client and thin client (distributed) topologies, as well as other possible configurations where the components are distributed across multiple devices or servers in the network.

In another aspect of the invention, a multi-modal shell comprises: a model manager for maintaining a dialog state of the application; a TAV (transformation/adaption/view preparation) manager for preparing and transforming pages or page snippets; and a synchronization manager for managing event notifications to the browsers. The components of a multi-modal shell can be distributed.

In yet another aspect of the invention, a WAP (wireless application protocol) multi-modal browser comprises:

a GUI (graphical user interface) browser comprising a DOM (document object model) interface for controlling the GUI browser and managing DOM and event notification and a wrapper interface for filtering events;

a speech application server comprising: a voice browser, wherein the voice browser comprises a DOM interface for controlling the voice browser and event notification and a wrapper interface for filtering events; an audio system for capturing and encoding speech data; and one or more speech engines for processing speech data; and

a multi-modal shell for parsing and processing a modality-independent application and managing synchronization of I/O (input/output) events between the GUI and voice browsers.

These and other aspects, features, and advantages of the present invention will become apparent from the following detailed description of the preferred embodiments, which is to be read in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a MVC framework according to, an embodiment of the present invention, which is preferably employed for building a multi-modal browser according to the present invention.

FIG. 2 is a diagram illustrating a programming framework for a single authoring programming paradigm which is preferably used for implementing a MVC-based multi-modal browser according to the present invention.

FIG. 3 is a diagram illustrating a relationship between various programming tiers of a MVC framework and single authoring programming model, according to one aspect of the present invention.

FIG. 4 is a diagram of an MVC-based multi-modal browser according to an embodiment of the present invention.

FIG. 5 is a diagram of an MVC-based multi-modal browser according to an embodiment of the present invention using a fat client approach.

FIG. 6 is a diagram of a distributed MVC-based multi-modal browser according to an embodiment of the present invention.

FIG. 7 is a diagram of a distributed MVC-based multi-modal browser according to an embodiment of the present invention.

FIG. 8 is a diagram of a distributed MVC-based multi-modal browser according to an embodiment of the present invention.

FIG. 9 is a diagram of a distributed MVC-based multi-modal browser according to an embodiment of the present invention.

FIG. 10 is a diagram of a distributed MVC-based multi-modal browser according to an embodiment of the present invention.

FIG. 11 is a diagram of a distributed MVC-based multi-modal browser according to an embodiment of the present invention.

FIG. 12 is a diagram of a distributed MVC-based multi-modal browser according to an embodiment of the present invention.

FIGS. 13a-13d are diagrams illustrating various methods of distributing a multi-modal shell according to the present invention.

FIG. 14 is a diagram of a distributed MVC-based multi-modal browser according to an embodiment of the present invention.

FIG. 15 is a diagram of illustrating a multi-modal shell framework according to an embodiment of the present invention.

FIG. 16 is a diagram of illustrating a multi-modal shell framework according to an embodiment of the present invention.

FIG. 17 is a diagram of illustrating a multi-modal shell framework according to an embodiment of the present invention.

FIG. 18 is a diagram of illustrating a multi-modal shell framework according to an embodiment of the present invention.

FIG. 19 is a diagram illustrating communication protocols that may be implemented with a multi-modal browser according to the present invention for coding, transport and control of audio data and control data.

FIG. 20 is a diagram illustrating communication protocols that may be implemented with a multi-modal browser according to the present invention for coding, transport and control of audio data and control data.

FIG. 21 is a diagram of a system and method for providing remote control of distributed speech engines which may be implemented with a multi-modal browser according to the present invention.

FIG. 22 is a diagram of a system and method for DOM implementation of a VoiceXML browser according to an embodiment of the present invention.

FIG. 23 is a diagram of a distributed WAP multi-modal browser framework according to an embodiment of the present invention.

FIG. 24 is a diagram of a distributed WAP multi-modal browser framework according to an embodiment of the present invention.

FIG. 25 is a diagram of a distributed WAP multi-modal browser framework according to an embodiment of the present invention.

FIG. 26 is a diagram of a multi-modal browser framework according to an embodiment of the present invention.

FIG. 27 is a diagram of a multi-modal browser framework according to an embodiment of the present invention.

FIG. 28 is a diagram of a multi-modal shell according to an embodiment of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present invention relates generally to systems and methods for building multi-modal user interfaces and applications, and in particular, to systems and methods for building modular multi-modal browsers based on a DOM (Document Object Model) and MVC (Model-View-Controller) framework that enables a user to interact in parallel with the same information via a multiplicity of channels, devices, and/or user interfaces, while presenting a unified, synchronized view of such information across the various channels, devices and/or user interfaces supported by the multi-modal browser

The following detailed description of preferred embodiments is divided into the following sections for ease of reference:

Section I below provides a general description of features and functionalities of a multi-modal browser according to the present invention, as well as the need, motivation and advantages of implementing a MVC and DOM-based multi-modal browser framework according to the present invention;

Section II describes preferred embodiments of a MVC architecture for building a multi-modal browser;

Section III describes various programming models that may be employed with a MVC-based multi-modal browser according to the present invention, wherein section III(A) describes single authoring programming paradigms and section III(B) describes multiple authoring programming paradigms;

Section IV outlines preferred features and characteristics of a multi-modal browser according to the present invention;

Section V generally describes various frameworks for building multi-modal browsers according to the present invention;

Section VI describes various architectures of multi-modal shells (or interaction managers) according to the present invention;

Section VII describes communication protocols that may be implemented in connection with a multi-modal browser framework according to the present invention, including, for example, conversational coding, transport and control protocols for encoding and transporting speech data and for control of distributed functions, as well as synchronization protocols for synchronizing information exchange between a multi-modal shell and various component browsers; and

Section VIII describes various exemplary embodiments of a multi-modal browser that can be implemented according to the present invention.

It is to be understood that the term "channel" used herein refers to a particular renderer, device, or a particular modality. Examples of different modalities/channels include speech such as VoiceXML, visual (GUI) such as HTML (hypertext markup language), restrained GUI such as WML (wireless markup language), CHTML (compact HTML), XHTML-MP and HDML (handheld device markup language) or any combination of such modalities.

The term "multi-channel application" refers to an application that provides ubiquitous access through different channels (e.g., VoiceXML, HTML), one channel at a time. Multi-channel applications do not provide synchronization or coordination across the views of the different channels.

The term "multi-modal" application refers to multi-channel applications, wherein multiple channels are simultaneously available and synchronized. Furthermore, from a multi-channel point of view, multi-modality can be considered another channel. As explained herein, the granularity of the synchronization may vary between sequential (i.e., suspend and resume mode), page level, block level (page fragments), slot level (gesture by gesture or dialog turn by dialog turn), event level/simultaneous and merged input (e.g., simultaneous voice and GUI input to time stamp and address as a single input).

Furthermore, the term "conversational" or "conversational computing" as used herein refers to seamless multi-modal dialog (information exchanges) between user and machine and between devices or platforms of varying modalities (I/O capabilities), regardless of the I/O capabilities of the access device/channel, preferably, using open, interoperable communication protocols and standards, as well as a conversational (or interaction-based) programming model that separates the application data content (tier 3) business logic (tier 2) from the user interaction and data model that the user manipulates. The term "conversational application" refers to an application that supports multi-modal, free flow interactions (e.g., mixed initiative dialogs) within the application and across independently developed applications, preferably using short term and long term context (including previous input and output) to disambiguate and understand the user's intention. Conversational application preferably utilize NLU (natural language understanding).

I. Motivation For Employing MVC and DOM-based Multi-Modal Browsers

A multi-modal browser framework according to the present invention is particularly advantageous for use with the Mobile Internet. Indeed, a value proposition for e-business solutions is to employ multi-modal applications/user interfaces/devices that allow users to: (i) enter and access data easily using small mobile devices (since, e.g., talking is easier than typing and reading is faster than listening); (ii) choose the interaction mode that suits the task and circumstances (e.g., input: key, touch, stylus, voice, output: display, tactile, audio, etc.); and to (iii) utilize several devices in combination (to thereby take advantage of the unique capabilities of each device). A multi-modal browser application according to the present invention provides seamless user interaction with multiple channels and devices. Indeed, it is expected that the mobile Internet will readily adopt user interfaces and applications that enable multiple, coordinated information channels—running either on the same or multiple devices or middleware—to be used simultaneously to gain sequential or parallel information access.

A multi-modal user browser according to the present invention allows a user to select an optimal interaction mode for each interaction between a user and an application. For example, stock charts or maps are more easily viewed as images, while complex queries are more effectively entered by voice. The choice of a particular interaction mode can be made by the developer of the application, or it can be left up to the user. For example, even if an interaction might be most effectively conducted via voice, a user may prefer to use stylus input if there are other people in the room. Similarly, even if an interaction is well-suited for a visual display and touch input, a user may prefer to use voice when his hands and eyes are busy. And a user who is interacting with an application by voice, say trying to arrange a flight while walking, may stop walking in order to interact visually when reaching a certain point in the application where he knows or feels that visual interaction is faster.

A multi-modal browser according to the present invention improves user interaction by allowing multiple, coordinated information channels—running either on the same or multiple devices or middleware—to be used simultaneously to gain sequential or parallel information access. A multi-modal browser framework according to the present invention provides a mechanism for parallel use of multiple access channels whereby transactions are shared across different devices. In addition, mechanisms are provided whereby updates to the underlying information via any given device or interface is immediately reflected in all available views of the information. A multi-modal browser (or multi-device browser) provides such coordinated, parallel user interaction by maintaining and utilizing shared application context and history, which enables all participating channels to share equally in the conversation with the user. The different channels provide similar and equivalent functionality while ensuring that the user is presented with consistent views of the underlying information that is being manipulated. In addition, interaction context and history is preferably synchronized across the various channels or devices so as to enable seamless transitions in the user interaction amongst the various channels or devices. Thus, user interaction with a specific device is reflected across all available channels; conversely, each available channel is primed to carry on the conversation with the user where a previously active device leaves off. This is closely related to the issues of transactional persistence/suspend and resume which, for example, enables a transaction performed on a PC to be interrupted and continued soon after by voice or WAP over a cell phone.

A multi-modal browser framework according to the present invention is applicable to multi-device applications and multi-channel applications. The synchronized and coordinated use of multiple devices in parallel will be especially important among pervasive clients. Today, users juggle between cell phones, pagers, PDAs and laptops. Synchronization mechanisms are provided but they merely guarantee that part of the information is shared and kept up to date across the devices.

An underlying principle of an MVC-based multi-modal browser according to the present invention is that a user participates in a conversation with various available information channels all of which communicate with a common information backend to manipulate a single synchronized model. The different participants in the conversation—including the user—will use the most appropriate modality to communicate with the target of the current portion of the conversation. Notice that when phrased as above, the role of the user and the various devices participating in the conversation is symmetric—a user can choose to point or use other visual gestures to interact with a particular device while using spoken commands to direct other portions of the conversation. The multi-modal interface driving the various devices can equivalently choose to display certain information visually while speaking other aspects of the conversation.

Key aspects of this form of interaction include the ability of a multi-modal browser to use the best possible combination of interface modalities based on the user's current preferences, needs and abilities as well as the application requirements and device capabilities. At the same time, the system is characterized by the ability to dynamically update its choice of modalities based on what the user chooses to do. Thus, upon failure of the user to respond to a spoken prompt, the system might choose to revert to a visual interface—an implicit assumption that the user is in environment where speech interaction is inappropriate—equivalently, a spoken request from the user might cause the system to update its behavior to switch from visual to spoken interaction.

Thus, a multi-modal browser application that is constructed in accordance with the present invention using mechanisms described herein advantageously support seamless transitions in the user interaction amongst the different modalities available to the user, whether such user interaction is on one or across multiple devices. When appropriate multi-modal user interface middleware becomes available, application developers and users will influence what information and under what preferred form is provided and acted upon in each modality. Automatic adaptation of the applications based on this consideration can be available on the server (application adaptation) or on the connected clients (user preferences, browser rendering features). A user interface according to the present invention supports dynamic and often unpredictable dynamic switches across modalities. Indeed, based on the user's activities and environment, the preferred modality can suddenly change. For example, a speech-driven (or speech and GUI) banking transaction will probably become GUI only if other people enter the room. Transactions that the user could not complete in his office are to be completed in voice only or voice only/GU constrained mode in the car. Preferred MVC frameworks that may be used to build multi-modal browser architectures according to the present invention are described, for example, in U.S. patent application Ser. No. 10/007,037, filed concurrently herewith on Dec. 4, 2001, (Express Mail Number EL797416039US), entitled "MVC (Model-View-Controller) BASED MULTI-MODAL AUTHORING TOOL AND DEVELOPMENT ENVIRONMENT", which is commonly assigned and incorporated herein by reference.

The DOM (Document Object Model) is a programming interface specification being developed by the World Wide Web Consortium (W3C) (see, e.g., www.w3.org). In general, DOM is a platform and language-neutral interface that allows programs and scripts to dynamically access and update the content, structure and style of documents. The document can be further processed and the results of that processing can be incorporated back into the presented page. With DOM, programmers can build documents, navigate their structure, and add, modify, or delete elements and content. Virtually, anything found in an HTML or XML document can be accessed, changed, deleted, or added using DOM.

In other words, the DOM is a programming API for documents. It is based on an object structure that closely resembles the structure of documents it models. The DOM is a logical model that may be implemented in any convenient manner such as a tree structure. Indeed, the name "Document Object Model" refers to the fact that documents are modeled using objects, and the model encompasses not only the structure of a document, but also the behavior of a document and the objects of which it is composed. In other words, the nodes of a DOM represent objects, which have functions and identity. As an object model, the DOM identifies: (i) the interfaces and objects used to represent and manipulate a document (ii) the semantics of these interfaces and objects—including both behavior and attributes (iii) the relationships and collaborations among these interfaces and objects.

The following is a brief detailed discussion regarding DOM, the use of the DOM, and how DOM can be implemented on user agents such as GUI user agents (e.g, WAP) and other pervasive devices. For the sake of discussion and by analogy to WAP, this DOM will be referred to as a DOM-MP (mobile profile).

DOM presents documents as a hierarchy of node objects that also implement other, more specialized interfaces. Some types of nodes may have child nodes of various types, and others are leaf nodes that cannot have anything below them in the document structure. Most of the APIs defined by the DOM specification are interfaces. A generic specification of DOM, including DOM level 1, 2 and 3 is known in the art and can be found, for example, in the reference by Arnaud Le Hors et al., Document Object Model (DOM) Level 3 Core Specification, version 1.0, W3C working draft, Sep. 13, 2001 (www.w3.org/TR/DOM-Level-3-Core).

The DOM provides a design of a generic event system that: (i) allows registration of event handlers, describes event flow through a tree structure, and provides basic contextual information for each event; (ii) provides standard modules of events for user interface control and document mutation notifications, including defined contextual information for each of these event modules; and (iii) provides a common subset of the current event systems used in DOM Level 0 browsers. This is intended to foster interoperability of existing scripts and content. The overall DOM L3 architecture is presented in the above reference. Each DOM level builds on the preceding level.

The following is a brief discussion of various DOM Interfaces. DOM Queries are interfaces that only enable access without modification to the DOM tree (structure and data). DOM Manipulation interfaces can artificially be more finely categorized as: (i) DOM Data manipulation interfaces that are used to access and modify data stored in the DOM tree without changing the structure of the tree; and (ii) DOM Tree manipulation interfaces that are used to access and modify the DOM tree structure. Moreover, DOM load/save provide interfaces for loading an XML document into a DOM tree or saving a DOM tree into an XML document, which includes many options to control load and save operations. Note that some functions can be fall under the categories of DOM data manipulation and DOM tree manipulation.

The DOM specification defines various types of events. For instance, User interface events are generated by user interaction through an external device (mouse, keyboard, etc.). UI Logical events are device independent user interface events such as focus change messages or element triggering notifications. Mutation events are events caused by any action which modifies the structure of the document.

The DOM describes various types of event flow. For instance, with the basic event flow, each event has an "event's target" toward which the event is directed by the DOM implementation. This "events target" is specified in the event's target attribute. When the event reaches the target, any event listeners registered on the event's target are triggered. Although all event's listeners on the events target are guaranteed to be triggered by any event which is received by that event's target, no specification is made as to the order in which they will receive the event with regards to the other event listeners on the event's target. If neither event capture nor event bubbling are in use for that particular event, the event flow process will complete after all listeners have been triggered. If event capture or event bubbling is in use, the event flow will be modified as described in the sections below. Any exceptions thrown inside an events listener will not stop propagation of the event. It will continue processing any additional events listener in the described manner. It is expected that actions taken by events listeners may cause additional events to fire.

Another event flow is Capturing, which is a process by which an event can be handled by one of the event's target's ancestors before being handled by the event's target. In addition, Bubbling is a process by which an event propagates upward through its ancestors after being handled by the event's target. Further, Cancelable is a designation for events which indicates that upon handling the event, the client may choose to prevent the DOM implementation from processing any default action associated with the event.

In accordance with the present invention, as explained in detail below, a DOM interface and associated mechanisms are preferably implemented with conventional browsers (such as WML and VoiceXML browsers) to provide support for browser control and event notification in a multi-modal browser using event specifications according to the present invention as described herein together with, e.g., DOM L2 event specifications. The sychronization granularity between the various views supported by the multi-modal browser will vary depending on the complexity of the DOM interface and synchronization mechanism that are supported. As will be apparent from the following detailed description of preferred embodiment, an MVC and DOM-based multi-modal framework according to the present invention provides an extremely modular and flexible approach to enabling various "fat" client and distributed "thin" client approaches using known network standards and frameworks.

II. Presenting Unified Information Views Via a Model-View-Controller Paradigm

FIG. 1 is a diagram illustrating a preferred programming paradigm for implementing a multi-modal application in accordance with the above-described concepts. A multi-modal application is preferably based on a MVC (model-view-controller) paradigm as illustrated in FIG. 1., wherein a single information source, Model M (e.g., a modality-independent representation of an application) is mapped to a plurality of Views (V1, V2) (e.g., different synchronized channels) and manipulated via a plurality of Controllers C1, C2 and C3 (e.g., different browsers such as a speech browser, a GUI browser and a multi-modal browser). More specifically, with this architecture, a multi-modal system comprises a plurality of Controllers (e.g., C1, C2, and C3) that act on, transform and manipulate the same underlying Model M to provide synchronized Views V1, V2 (i.e., to transform the single Model M to multiple synchronous Views). The synchronization of the Views is achieved by generating all Views from, e.g., a single unified representation that is continuously updated. A single Model M is transformed to multiple synchronous Views. These transformations can be inverted to map specific portions of a given View to the underlying Model M. Assume Ti denotes the mapping of the model M to the View i and T₁^-1 denotes the inverted mapping, composing Ti using Ti₁^-1 for appropriate values of i enables synchronization among the Views.

In other words, an MVC-based multi-modal system such as shown in FIG. 1 enables seamless switches between channels at any time, by continuously maintaining and updating the same state of the dialog in all interacting views, whether such channels comprise different devices or different modalities. A further consequence of the decision to embody multi-modal systems as collections of Controllers all of which manipulate the same underlying Model to provide synchronized Views, is that the system can be local (e.g. fat client) or distributed. This synchronization of Views is a direct consequence of generating all Views from a single unified representation that is continuously updated; the single modality-independent (channel-independent) representation provides the underpinnings for coordinating the various Views.

To see this, consider each View as a transformation of the underlying modality-independent representation and consider that the modality-independent representation is described in XML (declarative case). In this instance, the Model can be viewed as an abstract tree structure that is mapped to channel-specific presentational tree structures. These transformations provide a natural mapping amongst the various Views since any portion of any given View can be mapped back to the generating portion of the underlying modality-independent representation, and this portion consequently mapped back to the corresponding View in a different modality by applying the appropriate transformation rules.

Thus, in a preferred embodiment of the present invention, a multi-modal browser application is based on the MVC paradigm. The existence of a modality independent representation (authored or inferred) of the application enables implementation of the MVC, where the state of the application in that representation can be considered as the Model of the MVC architecture. More specifically, the Model of the interaction, which is independent of the rendering channel or modality, comprises a repository of the current dialog state, the dialog flow as currently known by the application and the whole conversation history and context when context management is needed. Any user interactions within a modality must act on the conversation Model before being reflected on the different Views.

III. Programming Models for Applications and MVC

It is to be appreciated that an multi-modal browser comprising a MVC framework such as shown in FIG. 1 can support either single or multiple authoring approaches. An MVC-based multi-modal browser according to the present invention can provide different levels of synchronization across the different modalities/channels/devices supported by an application, depending on the limits supported by the authoring method. For instance, a multiple authoring paradigm can support a given level of granularity, whereas a single authoring paradigm can advantageously support any level of synchronization.

A. Single Authoring

In general, "single authoring" refers to a programming model for authoring multi-modal applications, wherein the multi-modal application is specified in a representation that is independent of the target channel or modalities. Further, the modality-independent representation is specialized/optimzed for target devices or device classes. An underlying principle of single authoring is the Model-View-Controller, wherein the Model comprises a channel independent description of the application, each channel comprises a View of the model, and the Views are obtained by transforming the model representation into its target form which is rendered by Controllers such as channel specific browsers (e.g. WAP browser (WML or XHTML-MP), Web/HTML browser, C-HTML browser, HDML browser, VoiceXML voice browser, etc.). The user interacts with each View through a browser. Further, as multi-modality can be considered as a particular type of channel, the MVC principle becomes especially relevant for multi-modal or multi-device interactions. The user interacts via the Controller on a given View. Instead of modifying the View, his or her actions update the state of the Model, which results in an update of the different registered Views to be synchronized.

Accordingly, in a preferred embodiment of the present invention, a MVC-based multi-modal browser supports single authoring across a large variety of devices and modalities. Assume that "gestures" comprise units of synchronized blocks. For a single authoring method, gestures comprise elementary units defined by the language syntax and for which transformation rules are available for each View (channel). The Model (application) comprises a modality-independent representation that is dynamically transformed into the different channel specific languages. Naming conventions or node identification are associated to each of the resulting elements in each channel. Since any portion of any given View can be mapped back (through the node identification) to the generating portion of the underlying modality-independent representation, and this portion consequently mapped back to the corresponding View in a different modality by applying the appropriate transformation rules, the approach automatically satisfies the MVC principle.

Single authoring is motivated by the need to author, maintain, and revise content for delivery to an ever-increasing range of end-user devices. Generally, in a preferred embodiment, a single authoring programming paradigm enables separation of specific content from the presentation enabling reusable style sheets for default presentation in the final form. Specialization can then be performed in-line or via channel specific style sheets.

Single authoring for delivering to a multiplicity of synchronized target devices and environment provides significant advantages. For instance, as we evolve towards devices that deliver multi-modal user interaction, single authoring enables the generation of tightly synchronized presentations across different channels, without requiring re-authoring of the multi-channel applications. The MVC principle guarantees that these applications are also ready for synchronization across channels.

Such synchronization allows user intent expressed in a given channel to be propagated to all the interaction components of a multi-modal system. Multi-modal systems according to the present invention may be classified as "tightly-coupled" multi-modal interactions or "loosely-coupled" multi-modal interactions where each channel has its own model that periodically synchronizes with the models associated to the other channels. A tightly-coupled solution can support a wide range of synchronization granularities, as well as provide optimization of the interaction by allowing given interactions to take place in the channel that is best suited as well as to revert to another channel when it is not available or capable enough. The same approach can be extended to multi-device browsing whereby an application is simultaneously accessed through different synchronized browsers.

In a preferred embodiment of the present invention, an MVC-based multi-modal browser processes applications that comprise a single authoring programming framework that separates content, presentation, and interaction. For example, FIG. 2 is a diagram illustrating various programming layers comprising a single authoring programming model for implementing an application. A preferred single authoring model separates various programming layers comprising a backend data layer 20, a business logic layer 21, a data model layer 22, an interaction logic layer 23, a navigation layer 24, a specialization layer 25, and a modality-specific rendering layer 26. The business logic layer 21 is the portion of an application that contains the logic, i.e., encoded set of states and conditions that drive the evolution of the application, as well as variable validation information. In a preferred embodiment, the data models 22 (or data type primitives) are XML Schema compliant and defined in accordance with the proposed WC3 standard XFORMS Data Model (see, e.g., http:///www/w3.org/TR/xforms/). A modality-independent application preferably defines a data model, for the data items to be populated by the user interaction, and then declares the user interface that makes up the application dialogues.

The interaction layer 23 abstracts the application in terms of a finite set of interaction primitives (e.g., conversational gestures) to encapsulate the interaction logic in a modality-independent manner. One example of a preferred interaction language referred to as Interaction Markup Language (iML) will be explained in detail below.

The modality-specific presentation of the application as provided by the modality-specific presentation layer 26 is preferably based on the proposed XForms standard of separation of UI from the data models 22 (although the data model can be expressed using other suitable techniques). Lastly, the specialization layer 25 provides a mechanism for cosmetic altering a one or more features of a presentation, in one or more modalities. A default rendering of the conversational gestures depends solely on the gestures and the target modality or channel.

Separating content from presentation to achieve content re-use is a widely accepted way of deploying future information on the World Wide Web. In the current W3C architecture, such separation is achieved by representing content in XML that is then transformed to appropriate final-form presentations via XSL transforms. Other transformation mechanisms could be considered. A single authoring paradigm is particularly advantageous since in the near future, various embodiments of multi-modal browsers will be distributed. It will therefore be especially advantageous to support adaptation the granularity level of synchronization across the views to the network load or available bandwidth. Adaptation to the user's preferences or browser capabilities can also be supported.

Thus, the Model of an MVC framework according to the present invention preferably implements an application that is represented in a way that is independent of the target channel. Such representation abstractly describes the interaction and the data model that the user manipulates through it. At that level, the application is fully functional, independently of the modality or device where it will be rendered. Dynamic content and backend access to the business logical are conventionally programmed. The application can be transformed into presentations (final form) using default transformation rules that depend only on the target channel. Such presentations are defaults views of the applications adapted to the channel.

The application can now be specialized to specific channels or classes of channels. This can be done in-line or by specializing specific transformation rules. In particular such specialization can address the navigation flow, cosmetic layering and nature of the content finally presented to the user in each channel or channel class. Specialization of a fully functional channel-independent version of the application is a very efficient way to develop and maintain multi-channel applications.

An MVC framework according to the present invention is associated with the layer of an application that, in the 3-tier nomenclature, is conventionally called the presentation layer or tier 1 (and sometimes tier 0 when pervasive thin clients are introduced), as illustrated in FIG. 3. In FIG. 3, Tier 3 comprises the database (data) and an application to manage the database. Tier-2 comprises the business logic that runs on a Web application server, Web server, etc., which acts as a server to client requests. It is to be understood that the MVC concept of a modality independent representation of the application assumes that the conventional presentation layer (tier-1 and/or tier 0) is more finely factored and its boundary is somehow moved with respect to Tier 2 the business logic layer. FIG. 3 illustrates this issue, wherein Tier 2 overlaps Tier 1, Tier 0. Depending on the approach and programming methodologies, the correspondence between the various tiers can change.

In FIG. 3, it is assumed that a refinement of the decomposition into more tiers or layers and an implicit programming model for multi-modal applications guarantees the existence of a single modality/channel independent Model. With multiple authoring (as described below), this Model comprises a description of the synchronized blocks and their navigation flow. The Model needs to be extracted from the received ML page(s). Clearly, this Model depends on the type of modalities/channels to synchronize and issues like different prompts in different modalities or elements not addressed in a given modalities are addressed during authoring of the application. With single authoring, the Model describes the data model manipulated by the user and how this manipulation takes place (interaction). In one embodiment, the Model essentially comprises a DOM (Document Object Model) of the received page. This model, up to additional modality specific specializations, does not need to be aware of the type of modalities/channels to synchronize. Issues such as different prompts in different modalities or elements not addressed in a given modalities are taken care of at authoring during the specialization step.

Therefore, there is only one model and it must exist for the application that needs to be synchronized. But as indicated above, it will exist if the application is authored to support synchronization of different channels; by definition. Further, supported modalities do not affect the other tiers except for, e.g., the programming model or methodologies used to develop multi-modal applications and specializations that affect the business logic or data content (e.g. nature of the prompt).

Because there is only one model defined as above, it does not matter at the level of the model that the dialog will be by voice, GUI or a synchronized combination of the two. Therefore, if the two other tier layers of an application have been developed with this programming model in mind, then none of these layers should be affected either by the modality(ies) used.

So the model is not highly dependent on the used output media, but of course the resulting (synchronized) presentations are. There is only one application across the different tiers but it must be written to fit this programming methodology and the presence of an intermediate model. This can be achieved by various conventional and new approaches. In other words, the application is authored to reflect the differences in terms of the output presented to the user as generated by the 2nd tier for different modalities.

Of course, it is ultimately all a question of definition. If one considers that the Model of an MVC framework according to the present invention comprises the entire application across all tiers, then it could be considered that there is still one Model but it now dynamically adapts its behavior to the channels that it has to support and synchronize. But it is possible to see that as a set of different MVCs. Preferably, this view is not considered since it is preferable to, e.g., decouple a multi-modal browser from the backend tiers to prevent the multi-modal browser architecture from being directly tangled with the rest of the middle ware architecture and bound to middle ware and programming model choices that the web server provider or ASP may make.

A preferred embodiment of an interaction-based programming model that may be implemented in an MVC framework according to the present invention is described, for example, in U.S. patent application Ser. No. 09/544,823, filed on Apr. 6, 2000, entitled: "Methods and Systems For Multi-Modal Browsing and Implementation of A Conversational Markup Language", which is commonly assigned and fully incorporated herein by reference. In general, U.S. Ser. No. 09/544,823 describes a new programming paradigm for an interaction-based iML (interaction markup language) in which the application content (business logic and backend access) is separate from user interaction. More specifically, a IML programming model separates application programming into content aspects, presentation aspects and interaction aspects.

IML preferably comprises a high-level XML-based language for representing "dialogs" or "conversations" between user and machine, which is preferably implemented in a modality-independent, single authoring format using a plurality of "conversational gestures." Conversational gestures comprise elementary dialog components (interaction-based elements) that characterize the dialog interaction with the user and are bound to the data model manipulated by the user. Each conversational gesture provides an abstract representation of a dialog independent from the characteristics and UI offered by the device or application that is responsible for rendering the presentation material. In other words, the conversational gestures are modality-independent building blocks that can be combined to represent any type of intent-based user interaction. A gesture-based IML, for example, allows an application to be written in a manner which is independent of the content/application logic and presentation (i.e., gesture-based IML encapsulates man-machine interaction in a modality-independent manner).

Conversational gestures may be encoded either declaratively (e.g., using XML as indicated above) or imperatively/procedurally. Conversational gestures comprise a single, modality-independent model and can be transformed to appropriate modality-specific user interfaces, preferably in a manner that achieves synchronization across multiple controllers (e.g., speech and GUI browsers, etc.) as the controllers manipulate modality-specific views of the single modality-independent model. Indeed, application interfaces authored using gesture-based IML can be delivered to different devices such as desktop browsers and hand-held information appliances by transcoding the device-independent IML to a modality/device specific representation, e.g., HTML, WML, or VoiceXML.

In general, user interactions authored in gesture-based IML preferably have the following format: