Collaborative agent interaction control and synchronization system

Abstract

A system for supporting a coordinated distributed design process and which allows individuals to hold meetings over the internet and work together in a coordinated fashion on shared design problems is described.

Claims

What is claimed is:

1. A conferencing comprising:

(a) a plurality of participants;

(b) a like plurality of facilitator agents, each of said facilitator agents associated with a corresponding one of said plurality of participants, each said plurality of facilitator agents including:

(1) means for monitoring communication of a participant in the conferencing system;

(2) means for comparing the communication of the participant to a predetermined set of communication passages;

(3) means for providing a prompt to a participant in the conferencing system in response to the means for comparing finding a match between the communication of a participant and one of the predetermined set of communication passages; and

(4) means for presenting a caricature representation that informs the user of its current process state;

(c) a like plurality of social agents each of said social agents associated with a corresponding one of said plurality of participants, each of said social agents including:

(1) means for monitoring relevant components of a design environment and suggesting appropriate actions to be taken by a participant; and

(d) means for learning how to effectively represent each individual of a plurality of individuals in the design environment.

2. The conferencing system of claim 1 wherein said means for comparing the communication of the participant to a predetermined set of communication passages includes means for learning communication passages by being presented with one or more situation-action pairs which are classified in a rule-base such that said facilitator agent can proactively respond to future situations without user intervention.

3. The conferencing system of claim 1 wherein said means for monitoring relevant components of a design environment and suggesting appropriate actions to be taken by a participant includes means for monitoring and suggesting by being presented with one or more situation-action pairs which are classified in a rule-base such that said social agent can proactively respond to future situations without user intervention.

4. The conferencing system of claim 1 wherein said social agents express an emotion of a participant.

5. The conferencing system of claim 1 wherein said facilitator agents suggests a change of forum in response to a particular agenda item.

6. The system of claim 1 wherein said facilitator agent further comprises means for evolving a reaction with the use of the conferencing system.

7. The system of claim 1 wherein said facilitator agent further comprises:

means for coupling to a control mechanism; and

means for establishing an appropriate control mechanism for a given forum setting.

8. The system of claim 1 wherein at least one of said facilitator agents further comprises means for building a rapport with a conferring participant through an interface technique that builds trust between the participant and each of the at least one of said facilitator agents.

9. The system of claim 1 wherein said facilitator agent farther comprises:

means for presenting a choice to a participant; and

means for accepting a decision from the participant.

10. The system of claim 1 wherein the process state corresponds to one of thinking, suggesting, gratified, disappointed, or confused.

11. The system of claim 1 further comprising emotion representing means for providing a suggestion in response to a decision and for showing expressions of sadness or happiness dependent upon the reaction of the participant.

12. The system of claim 1 further comprising means for building thresholds for decisions that may be taken without user intervention.

13. The system of claim 1 further comprising means for encoding user preferences for agent autonomy such that said agent includes an indication of what decisions the participant with which the agent is associated is willing to delegate to the agent.

14. The system of claim 1 wherein the agent's decisions on process interventions are based on the following aspects of the agent environment:

(a) a current topic's recommendation list;

(b) threshold levels indicating user preferences;

(c) an averaged record of the participation of each participant in a negotiation process; and

(d) a complete conversation model of an ongoing negotiation.

15. The system of claim 1 further comprising:

means for generating a vector representing the averages of the on-line time consumed by each participant in a forum;

means for generating a vector representing the amount of recommendations generated regarding a specific intent;

means for generating a vector representing the average time a participant in a forum waits before given the right to speak; and

means for generating a vector representing the relevance of a topic to each participant in a forum.

16. The system of claim 15 wherein a weight is assigned to each of the vectors and the elements are summed.

17. The system of claim 16 wherein the weights are adjustable by a user.

18. The system of claim 1 wherein the facilitator agent distinguishes between: (1) brainstorming/free control strategy, (2) a lecture control strategy; and (3) a chairperson control strategy.

19. The system of claim 1 wherein said social agent further comprises:

means for becoming more familiar with a participant with which the agent is associated;

means for cooperating with the agents of other participants in the conferencing system to decide upon an appropriate meeting control scheme; and

means for reaching a consensus on a meeting process intervention.

20. The system of claim 1 wherein at least one of said facilitator agents further comprises:

means for becoming more familiar with a participant with which the agent is associated;

means for cooperating with the agents of other participants in the conferencing system to decide upon an appropriate meeting control scheme; and

means for reaching a consensus on a meeting process intervention.

Description

GOVERNMENT RIGHTS

Not applicable.

FIELD OF THE INVENTION

This invention relates to systems for supporting a coordinated distributed design process and more particularly to systems which allow individuals to hold meetings over the internet and work together in a coordinated fashion on shared design problems.

BACKGROUND OF THE INVENTION

As is known in the art, collaborative systems refer to a class of systems in which multiple users participate in a particular activity (e.g. a business meeting, a meeting to solve an engineering problem, a lecture, etc. . . . ).

As is also known, the emergence of high speed communication networks and improved visualization techniques has laid the foundation for making such computer based collaboration practical. Various collaboration tools and conferencing systems have been developed by academic institutions, office system manufacturers and communication companies.

The literature is rich with research in the area of computer mediated communication. The work spans multiple disciplines and three diverse focus areas have been identified as emerging in this research field: (1) Electronic Meeting Systems(EMS); (2) Video Conferencing; and (3) Shared social spaces. Each of these groups represents a different approach to computer mediated communication.

EMS research focuses on the meeting process and decision support tools for the meeting process. Video conferencing research is concerned with transmitting multi-media data between participants (esp. audio and video data). The shared social spaces perspective is concerned with enabling interaction and experience across distance and providing awareness and persistence within a virtual world.

Electronic meeting systems encompasses a large body of research dedicated to the support of participants in traditional meeting settings. The GroupSystems EMS and the Xerox Parc Collab project are among the first such systems developed. Both systems have tools that structure brainstorming and problem solving processes and enforce interaction controls on the participants within the shared media. However, the control of floor in discussion is governed by regular meeting norms since all participants are co-located. It has been found that some of these additional process structuring constraints on the collaboration are not necessary and may decrease satisfaction within the workgroup.

Initial research on video conferencing focused on the technical aspects of transmitting video and audio data among individuals. Much of the initial work was constrained to two-person interactions and a large portion of the work utilized a telephony paradigm for the interaction. Further developments have occurred rapidly in this field and most modern systems such as Microsoft NetMeeting, Intel Proshare, PictureTel, and SGI Inperson provide multi-person interaction and have extended audio and video services to include shared whiteboards, editors and browsers. However, these conferencing systems lack any appropriate concurrency control mechanisms and are cumbersome to use for group work.

The final area of research in telepresence is devoted to the study of virtual communities and interaction in a virtual environment. Several tools have been developed to provide awareness, persistence, and interaction in cyberspace. The two leading research efforts in this field are TeamRooms and Worlds. The primary concern of these systems is in the representation of physical concepts of space and place in the virtual world. The environments developed provide richer interaction contexts, but are currently constrained by network bandwidth and display technology.

It would, therefore, be desirable to provide a system which allows individuals to hold meetings over the internet and work together in a coordinated fashion on shared design problems. It would also be desirable to have a system which provides automated facilitation services, which supports a variety of meeting structures and which provides floor control policies to dynamically control the transition of the speaking state (the floor) from one participant to another (floor control policies).

SUMMARY OF THE INVENTION

In accordance with the present invention, a collaborative agent interaction control and synchronization system of the present invention interprets a physical meeting environment and represents the physical meeting environment as a virtual environment and also exploits the characteristics of the communication medium. This is accomplished through the deconstruction of group interaction into its core elements and the translation of these elements into computational representations. In addition, facilitation processes have been modeled in order to allow intelligent agent manipulation of the meeting process. With this particular arrangement, a system for allowing individuals to hold meetings over the internet and work together in a coordinated fashion on shared design problems is provided.

In accordance with a further aspect of the present invention, a facilitator agent for use in a conferencing system includes (a) means for monitoring communication of a participant in the conferencing system; (b) means for comparing the communication of the participant to a predetermined set of communication passages; and (c) means for providing a prompt to a participant in the conferencing system in response to the means for comparing finding a match between the communication of a participant and one of the predetermined set of communication passages. With this particular arrangement, a system which provides automated facilitation services and supports a variety of meeting structures and floor control policies is provided. Each participant in the conferencing system has an associated facilitator agent. The facilitator agent is able to recognize communication between two participants e.g. via training with action-pairs. By being able to recognize the communication, the facilitator agent is able to prompt the participant with certain suggestions. For example, if a participant in a brainstorming or free style discussion taking place around a round table types or otherwise communicates (e.g. with audio or visual cues) the phrase "let me explain," the facilitator agent may prompt the participant by asking the participant if he wishes to switch to a lecture hall setting which may be a more appropriate setting in which to give an explanation. Also, the facilitator agent may recognize during a meeting that an allotted amount of time has been spent on a particular agenda item. In this case, the facilitator agent may prompt one of the participants (e.g. the chairman of the meeting) by asking if the chairman would like to move on to the next agenda item.

In accordance with a still further aspect of the present invention, a social agent for use in a conferencing system includes (a) means for monitoring relevant components of a design environment and suggesting appropriate actions to be taken by a participant and means for learning how to effectively represent each individual of a plurality of individuals in the design environment. With this particular arrangement, a social agent which can represent an emotion or other social interaction characteristic is provided. When the social agent detects one or more communications which it recognizes as corresponding to an emotion of the user, the social agent can suggest to the user an appropriate cue in order to reflect the users emotional state. For example, the social agent may provide a visual cue by utilizing an icon of a person with a happy or sad face or by changing a color associated with that user. Thus, the social agent can suggest an image (e.g. an icon) or other cue (e.g. audio) which represents the emotional state of the user (e.g. happy, sad, satisfied, concerned, etc. . . . ).

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features of the invention, as well as the invention itself may be more fully understood from the following detailed description of the drawings, in which:

FIG. 1 is a block diagram of an interaction system;

FIG. 2 is a diagrammatical representation of a hierarchy of forums;

FIG. 3 is a block diagram of a collaborative agent interaction control and synchronization system;

FIG. 4 is a diagram of agent architecture;

FIG. 5 is a diagram of a facilitator agent;

FIG. 6 is a diagram of an agenda tool and forum manager interface;

FIG. 7 is a view of a chairperson mode;

FIG. 7a is a view of a lecture mode;

FIG. 7b is a view of a brainstorming mode;

FIG. 8 is a diagram of a social agent;

FIG. 9 is a diagram of a collaboration mechanism in a system such as that shown in FIG. 3;

FIG. 10 is a block diagram of a synchronization system between two conference participants is a system such as that shown in FIG. 3;

FIG. 11 is an object diagram of a control infrastructure;

FIG. 12 is a diagrammatical representation of media frame synchronization;

FIG. 13 is a block diagram of a real time scheduler;

FIG. 14 is an image representing a user interface including a control panel;

FIG. 15 is a diagrammatical view of a meeting entry/exit interface

FIG. 16 is a diagrammatical view of a meeting with side conversations;

FIG. 17 is an image representing a speaking request;

FIG. 18 is an image representing a speaking request as seen by a meeting chairman;

FIG. 19 is an image representing a chairman granting a request to speak;

FIG. 19A is an image representing who is talking;

FIG. 20 is an object diagram showing server interconnectivity;

FIG. 21 is an object diagram of an Agenda Editor and Wizard classes;

FIG. 22 is an object diagram of the structure of the participant in a conferencing system of the type described above in conjunction with FIG. 3;

FIG. 23 is an object diagram of media driver classes;

FIG. 24 is an object diagram showing a queue structure hierarchy;

FIG. 25 is a flow diagram for the forum process;

FIG. 26 is a flow diagram for a speaking request;

FIG. 27 is a block diagram of a conferencing system operative over a computer network;

FIG. 28 is a block diagram of an illustrative computer system on which may be executed computer code to allow participants to use a collaborative system.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Terminology

Before describing a Collaborative Agent Interaction Control And Synchronization System and the operations performed to allow individuals to hold meetings over the internet and work together in a coordinated fashion on shared design problems, some introductory concepts and terminology are explained.

An "Agenda" is a set of guidelines for the topics of discussion in a particular meeting setting. "Asynchronous" interaction is communication that is stored in some form before transmission to the receiver of the information.

"Collaboration" refers to work performed jointly with others especially in an intellectual endeavor. "Conferencing" is the act of structured and formal group interaction and sometimes refers exclusively to distributed group interchange.

"Facilitation" is the act of providing process interventions in group discussion to enhance the efficiency and effectiveness of the interaction.

"Floor" refers to the right of one member of a group to communicate to other members within a group discussion or meeting.

"Interaction" is communication that engenders reciprocal communication or action.

"Knowledge" refers to the conceptual model of the environment and other information that a person is exposed to and assimilates.

"Learning" is the process by which knowledge is built, transferred and shared.

"Meeting" refers to individuals coming together for a common purpose.

"Negotiation" refers to the process of resolving conflicts or disputes among individuals.

A "participant" refers to a user of the conferencing or collaboration system that registers with a Name Server. A participant may have a specific role in a collaboration instance. The simplest example is that of a chairman. The chairman is in charge of a meeting and presides over the proceedings. On a deeper level, there may be participants that are in charge of certain functional units. He/she may have specific applications or databases that are dedicated towards his/her role and can bring any of these to a session.

"Synchronous" refers to events happening, existing, or arising at the same time while "synchronous interaction" refers to communications that are immediate and whose expected response is immediate. These include face to face meetings, telephone calls and video conference interactions.

"Telepresence" is the ability to provide a semblance of co-location of distributed individuals through the use of computer and communication technologies.

Referring now to FIG. 1, an interaction system 10 includes a plurality of interaction environments 12a-12N, generally denoted 12. Each of the interaction environments include a processing device or machine coupled via a network 14 over which an interaction protocol 16, a communication protocol 18 and interaction modes 20 are implemented and enforced between the environments 12. The interaction environments 12 define an interface between human and machine models of the interaction. The interactions are typically exemplified by metaphors that represent protocol distinctions. For example a room metaphor is commonly used to denote presence of individuals in proximity. The interaction environments 12 thus define a space for the individual's interface to the machine and other networked individuals.

The communication protocols 18 correspond to a set of rules for information transmission across a network and enable the transmission of information from one machine to another through the network 14.

Interaction protocols 16 enforce order on the communication over the network collaboration by controlling the ability to address particular individuals. An interaction protocol is a set of rules and algorithms that govern the accessibility of other participants in an interaction. These include definitions of proximity (proximity in a distributed sense has different implications than in the physical environment), addressability (controls over the ability to interact with others in the interaction environment) and presence (definitions of visibility of individuals to others).

The interaction modes 20 define the input and output devices by which information is displayed within each individual's interaction environment. Interaction Modality defines the variety of information structures and media available to the interaction. These may include audio transmission, video transmission, image transmission, text transmission and structured data (in the form of databases, schedules, CAD drawings, formatted text, etc. . . .)

Referring now to FIG. 2, a forum 22 which maintains a conference among individuals is comprised of other forums 24, 26 as well as a plurality of individuals 28a-28e. Each forum is associated with a forum moderator that defines the control behavior of the conference. The associated forum server (e.g. one of forum servers 34a, 34b in FIG. 3) processes requests for membership to the forum as well as requests to speak by participants within the forum. As shown in FIG. 2, the Management Forum 26 is a member of another forum called Project Forum 24. Thus, the forums 24, 26 and 28a-28e are arranged in a hierarchical forum structure.

The project and management forums 24, 26 have a parent-child relationship and any restrictions on membership and communication must be upheld by the child forum (i.e. management forum 26). Thus, the Management Forum 26 must be at least as restrictive as the Project Forum 24.

Referring now to FIG. 3, a collaborative agent interaction control and synchronization system or more simply a collaboration or conferencing system 30 includes a name server 32 having a pair of forum servers 34a, 34b coupled thereto.

Each participant (a)-(d) engaged in a conference spawns a collaboration manager (shown as a dashed boxes 36a-36d) which is comprised of media drivers 38, 40, 42, (shown in FIG. 3 as pictograms of the media i.e. video camera, microphone and X display) and message servers 44 (indicated by the acronym 'MSG'). The operation of the media drivers 38, 40, 42 is described in further detail below. The message servers 44 package data for transmission over the network and enforce synchronization constraints during media play-back.

The forum servers 34A, 34B are processes that maintain control of a conference among several individuals and enforces membership constraints. Furthermore the forum servers 34a, 34b log all conference proceedings. Forum servers 34a, 34b are spawned by forum managers (not shown) that define a specific control methodology. Forum managers 36a-36d also provide mechanisms for converting a forum server's control strategy. Finally, the name server 32 maintains a directory of all participants, forum managers and forum servers within the conferencing/collaboration system 30. The name server 32 allows each participant (a)-(d) to easily address any other member or forum in the collaboration system 30.

The name server 32 is an independent server that acts as a global directory for the conferencing system 30. The following information is listed in the name server for each participant and each forum and may be queried by any participant or forum server: (1) Participant Name and Location: including media driver locations and media descriptors; (2) Participant Status: each participant is either in an active or non-active state. Active denotes that the user is logged into the conference system via a Collaboration Manager on his/her workstation. Non-active status is given to users who are subscribers to the CAIRO system but are not reachable; (3) Forum Manager Name and Location: including a brief description of control style; (4) Forum Name and Location: including a listing of shared media drivers. (5) Forum Status: each forum is either in an active or non-active state. Active forums imply a conversation is occurring among the participants of the forum. Non-active forums are structured meeting skeletons with membership lists for a meeting that is not currently in session.

During a meeting or conversation, a particular participant can be in one of three states: active (i.e. speaking or demonstrating), pending (i.e. awaiting his/her turn to speak), or inactive (i.e. passive observer or listener). Each participant's state is relative to a specific forum and is stored in the forum server.

A speaker may initiate a request (a speaker request): This is equivalent to a professional raising his/her hand in a meeting situation. The speaker request indicates to the forum moderator and to the other members of the forum the participant's intent to speak. A speaker request is accompanied by a qualification of the speech act the speaker intends to perform. The forum server would then place the participant on a list of pending speakers depending on his/her qualifications. In a democratic forum a participant becomes active if a majority agrees to his/her request to speak. Furthermore, the interaction facilitator agent process executed on the server can automatically moderate (i.e. choose the active speakers from the pending queue) a forum based on pre-compiled speaker qualification data. The system includes a mode of conversation referred to as "interjection" in which the participant can interrupt an ongoing conversation for a limited amount of time.

Group meetings can take on multiple characters and structures and thus group formafion and meeting cycles require various group control procedures and paradigms. A list of primitive controls on each forum from which a more complex collaboration control mechanism may be devised includes: (1) chairperson, (2) inteljection duration, (3) maximum speech duration, (4) maximum number of active speakers, (5) side conversations, (6) external conversations, (7) logging mode, (8) speaker evaluation, (9) speaker ordering, (10) sample collaboration schemes and (11) completely uncontrolled. The forum creator may choose to over-ride any of these primitives for a particular forum member.

In the chairperson mode, one or more participants—each referred to as a chairperson, are designated as holding a privileged status within the forum. The chairperson may preempt speakers and arbitrarily choose active speakers.

The interjection duration control specified for a forum is the length of time allowed for interjections. An interjection time of zero indicates no interjections are allowed. Conversely an infinite interjection time allows for complete unstructured free-form conversation.

A maximum speech duration control specified for a forum is the length of time allocated to a single member to hold the floor of the conference while the maximum number of active speakers control parameter indicates the number of concurrent speakers allowable during the conference.

The side conversations control enables/disables side conversations. Side conversations are two-way or multi-way conversations among a subset of the forum members. Thus, forums may be created that do not allow such side conversations to exist.

Similarly, external conversation control parameters enable/disable the ability to have external conversations. External conversations are conversations between a member of a forum and other non-members while a forum is active. This form of conversation may also be restricted by the forum.

The logging mode allows either continuous logging or no logging at all of the ongoing conference. The speaker evaluation control provides a voting mechanism implemented to evaluate participant acceptance of a specific topic or to determine participant value to a conference. The results of this evaluation may be used to determine the order of speaker priority for a conference. The speaker ordering control allows the ordering of the pending speaker queue. This may be on a first come first serve basis or it may be based on some other evaluation criteria including but not limited to ordering of speakers based on value determined by the participants (as described in conjunction with the speaker evaluation control) or ordering based on chairperson choice in a chairperson controlled conference. This control mechanism satisfies the requirement for free form and structured conferencing

The collaboration primitives discussed above are combined to form a collaboration scheme or mechanism. The conferencing system of the present invention can easily be extended to provide many different collaboration schemes. The list of schemes which can be implemented includes but is not limited to: (1) a free scheme in which all participants may talk at any time; (2) a completely uncontrolled scheme in which all speakers may speak at once; (3) a democratic scheme in which the choice of the active speaker is based on a vote by all other participants; (4) a chalk-passing scheme in which the last active speaker chooses the next person to be a designated active speaker and each speaker may only speak for the time allotted by the Maximum Speech Duration parameter specified above; (5) a chairperson control scheme in which a specific privileged participant (Mr. X) has the ability to choose the participant who should address the conference at any specific time; (6) a Modified Delphi scheme in which the system polls all participants in the collaboration on their views regarding a specific design problem and the results are compiled and presented to the conferring experts and the participants are then re-polled. This process is repeated by the questioner until the experts provide a consistent analysis. The Delphi method is used extensively in polling experts on directions in hi-tech industry. In this control strategy there exists a moderator as well as a questioner.

The completely uncontrolled scheme, for example, can be specified with the following parameters: chairperson='none', side conversation=all, external conversation=all, speaker ordering=first-come-first-serve. The democratic scheme can be specified with the following parameters: chairperson='none', side conversation=all/none, external conversation=all/none, speaker ordering='highest vote'. The chalk-passing scheme can be specified with the following parameters: chairperson='last speaker', side conversation=all/none, external conversation=all/none, speaker ordering='chosen by chairperson'.

The chairperson control scheme can be specified with the following parameters: chairperson='Mrs. Q', side conversation ALL/NONE, external conversation=ALL/NONE, speaker ordering='chosen by chairperson'.

Furthermore, a scheme which is relatively quick and more dynamic compared with prior art techniques can be implemented using the collaboration techniques of the present invention. This scheme can be specified with the following parameters: chairperson='moderator/questioner' side conversation=all/none, external conversation=all/none, speaker ordering='round robin'.

Referring now to FIG. 4, an agent architecture includes a forum 40 having an associated agenda 42. Also associated with the forum 40 are a plurality of clients 44a-44c. Although three clients are here shown, it should be appreciated by those of ordinary skill in the art that any number of clients (limited only by the current technological state of the art), may be used. Each of the clients 44a-44c, has an associated agent 46a-46c. Each of the agents 46a-46c provides information to its respective participant-client 44a-44c depending upon information concerning the participants 44a-44c received by the agent The class for the implementation of the agent suggestions is shown in Appendix A.

FIG. 4 illustrates the agents that are involved at the forum level and at the client interface. The facilitator agent keeps track of a meeting agenda and also the meeting process. The agent at the client interface keeps the user informed about the meeting participants and bridges the user interaction with the rest of the meeting members with social queues.

When a user wishes to use the system, he or she must start up the client 44 and register with the nameserver 32 (FIG. 3). In one embodiment, the clients 44a-44c and nameserver 32 are Java message sending programs executing on some machine coupled to the internet. Once a client 44a-44c has registered with the nameserver 32, the nameserver 32 provides the client 44a-44c with a list of all the meetings that are currently running on the conferencing system 30 (FIG. 3). A forum 40, also referred to as a meeting, is another Java message sending program. Once a client (e.g. one of clients 44a-44c) decides to enter a forum (e.g. forum 40), messages are then passed directly from the client to the particular forum.

A forum is a virtual presentation of a meeting room (e.g. meeting rooms 90-94 in FIGS. 7, 7A and 7B) where clients can collocate over the internet. The basic concept is much like a chat room, where users can join a chat and talk to each other. One difference, however, is that the conferencing system of the present invention provides communication tools to the clients 44a-44c that are more conducive to the engineering and other collaboration processes. Another distinction between a chat room and the conferencing system of the present invention is that the conferencing system of the present invention embodies the notion of a meeting control strategy, including chaired meetings, roundtable meetings, and lecture meetings.

Furthermore, chat rooms allow anyone to comment or talk at any point in time. However, actual engineering negotiation meetings cannot follow this format for it would lead to chaos. Thus, in contrast to a chat room, the conferencing system of the present invention allows forums to take on characteristics similar to those of actual meetings. The characteristics are implemented using so-called meeting control strategies. A meeting control strategy is a way of deciding which member of a meeting is allowed to contribute to the meeting. An example of a meeting control strategy is one that models a chaired meeting. In the chaired meeting strategy, one client is designated as the chairperson of the meeting. If another client wishes to collaborate with other clients in the meeting, he or she must first request permission from the chairperson. Another example is a roundtable control strategy. As may be inferred, any client can collaborate at any point in time for there is no chairperson, and permission need not be requested.

To help implement the meeting control strategies, the collaborative system utilizes behavior based learning agents. A behavior based learning agent includes means for analyzing group and individual performances, means for determining user preferences, a conferencing means, a learning means coupled to the conferencing means for receiving information over a period of time and for updating responses to the learning means over a different period of time through interaction with a user and the conferencing means. The learning agent also includes means for interacting with another user representative agent and for agreeing on an appropriate process intervention to enhance a meeting process.

The learning means learns by being presented with one or more situation-action pairs which are classified or structured in a rule-base such that the learning means can proactively respond to future situations without user intervention.

Behavior based learning agents are used to perform the analysis of group and individual performance as well as to determine user preferences. The agent collects information on a variety of user parameters including but are not limited to: amount of time on the pending speaker queue, number of people on the pending queue, qualifications of the speaker/audience and their relevance to the agenda topic, activity on each member computer/system, level of engagement of members of the conference, a brief history of level of interaction of each individual (including but not limited to time spent as speaker, time spent on the pending queue, number of interactions with other users etc. . . . )

These agents learn over time through interaction with the user and the conferencing application. The agent learns to represent the user and express his/her frustrations. All user representative agents can then agree on an appropriate process intervention to enhance the meeting process. A user representative agent maintains information about the behavior of a single member in the conference. It maintains the history of the interaction in the conference, a description of the interests/expertise of the user, a history of past participation in conferences (time spent, time spent speaking, time spent pending etc. . . . ). They encode all data on a single user across a set of meetings. The interaction facilitator agent on the other hand retains Information on a single meeting without historical data on each of the participants.

Agents learn by being presented with situation-action pairs which are classified or structured in a rule-base. Situation-action pairs are rules, each of which has a situation and a corresponding action. Each agent has a set of situation-action pairs that it uses as a guide for the facilitation process. When encountered by a particular situation the agent looks up its set of rules and acts accordingly. An example of this would be the dictionary that the agent uses to monitor user cues from the textual chat that the user inputs. In this case the agent parses smilies (textual representations of expressions such as smiling:-) or frowning:-(or other expressions) that the user types to represent an emotion graphically. Future situations can then be proactively responded to without user intervention.

Two design decisions in agent implementation are feature selection and learning algorithm choice. There are several features that constitute the whole learning process of user preferences. These features can be listed as follows: (1) User modeling and (2) Learning about the user.

User modeling refers to the capability of methodology and the system to capture information pertaining to the context of collaboration. The main components of this are Knowledge representation which defines the internal structure and functionality of the user model and Persistent profiling which defines how to reuse a user model over time and across applications.

Learning about the user refers to the process t performed in order to better fine tune and to automate the development of the user model. In order to achieve this a method of machine capturing and generalization of data is done using the Data observed and learned, a Generalization Scheme and a Memory model.

In one approach, to increase the effectiveness of agent personification and in building user acceptance, it may be desirable to utilize "dumb" agents that learn through interaction with the user in order to build a relationship between the agent and the user.

As mentioned above, "dumb" agents build trust incrementally with the user. On the other hand, "dumb" agents may also be very frustrating and ineffective in simplifying the user's tasks. They may require additional boot-strapping time before becoming effective in simplifying user tasks.

It should be appreciated that the prior art techniques can be used to train the dumb agent. The core idea is to build a learning engine that uses a particular learning technique referred to as clustering algorithms. These techniques have been implemented in order to train an agent over a short term as well as a long term. This is where the issue of short term and long term memory is dealt.

Referring now to FIG. 5 a facilitator agent 50 is shown. The facilitator agent 50 is a program that monitors the actions of the users, residing with the moderator of the meeting. The agent 50 procures information from the interactions among users and tries to make intelligent suggestions to the moderator that may increase efficiency in the meeting proceedings. The agent possesses functionality, focusing on the agenda, certain textual cues, and request activity.

For example, during a meeting, if the time allotted for an agenda item has expired, the agent will appear and provide a prompt 52. In this example, the agent 50 prompts the moderator to determine if the moderator wishes to continue to the next agenda item. In this case the prompt 52 is a text prompt asking "Next agenda item?" It should be appreciated, however, that in some embodiments different prompts may be used including but not limited to audio prompts and color prompts.

In this case, the agent 50 is a passive timer for the user. The client can choose to accept the suggestion or reject it depending on how he/she feels the meeting is proceeding by "clicking" on the accept or reject buttons 54, 56 respectively. However, the agent 50 still serves as a reminder that keeps the participants on topic and in-line with their meeting agenda.

Referring now to FIG. 5A, the agent 50 also reacts to certain textual phrases that are sent and received through the text driver. The agent 50 includes functions to parse the text that is being delivered for specific key words. For example, if the agent 50 recognizes a word such as "explain" 58, the agent 50 may suggest to the moderator that he/she change the meeting style from the free style view 60 to a lecture control strategy (e.g. by changing the view to a lecture hall 92 shown in FIG. 7A).

Similarly, if the word "discuss" were recognized by the agent 50, then the agent 50 would suggest that the freestyle meeting strategy 60 be employed. Again, the user can then decide whether to accept the suggestion or not.

Finally, the agent 50 also responds to actions that are being taken by the users. If the agent 50 notices that a lot of requests to speak are being made by the participants, then the agent 50 may recommend that a freestyle meeting strategy be applied (i.e. that a free style view such as view 60 be used). This way, the meeting may run more smoothly if users do not have to constantly request permission to speak.

The meeting control process should help dissipate conflict elements among the participants during an engineering design negotiation. Negotiators often employ professional facilitators to invoke meeting process changes to ensure the focus of the negotiating group and to reduce conflict. Characteristics of effective facilitators include but are not limited to: (1)neutrality; (2) process expert; (3) maintains issue focus; (4) information resource; (5) trustworthy, non-authoritative; and (6) not a decision maker.

In the system of the present invention, some of these requirements are satisfied through the use of agents such as the facilitator agent 50. These agents reduce the overhead required in the facilitation of change negotiation. The facilitator agents are: (1) inherently neutral since they are computer-based and have no emotions or agendas; (2) encoded with various process intervention heuristics and their reaction evolves with the use of the system; and (3) trustworthy since they acquire the trust of the user by informing the user of all the agent's decisions and allowing the user to adjust agent control parameters as they evolve.

The facilitator agents are coupled with several control mechanisms described herein. The control mechanisms are different meeting styles and the protocols that go with each kind of meeting style. The different meeting protocols that have been implemented are Chairman style, Lecture style, Class room style, brainstorming and the Court system.

The agents establish the appropriate control mechanism for a given meeting setting. The choice of conference control strategy (e.g. freestyle, lecture or board meeting) at appropriate meeting milestones is critical in the effective coordination of group effort. The agent architecture developed in the conferencing system of the present invention automatically determines the strategy relevant to the current topic of discussion.

The basis of the agents decision is an encoded meeting agenda as well as the meeting indices. Although the agenda and process representations are linear, typical design discussions involve a cyclical refining process. Hence, the agent must be able to traverse the agenda and process in accordance with the discussion patterns. The mapping of process and agenda stage to appropriate facilitation strategy are currently very simple and are based on several heuristics. The critical issue is the agent's responsibility to determine the transition between the stages outlined in the agenda and process model. The agent bases its behavior on the conversation graph and its relation to the design process and expressed agenda.

Referring briefly to FIG. 6 a screen image 80 provided by an agenda tool includes an forum manager portion 82 and agenda portion 84. The forum manager portion 82 includes has fields which specify the meeting date 82a, meeting length 82b, person calling the meeting 82c, person responsible for the minutes 82d, the type of forum 82e, the subject 82f, the purpose 82g, the type of meeting 82h, and RSVP regrets e-mail address 821, an email address 82j and a copy line 82k.

The agenda portion 84 includes a first agenda item 84a, "system architecture," a second agenda item 84b "Communication Protocols" and a third agenda item 84c "Naming Conventions." By selecting an agenda item and selecting the "View" button, users can see who is in charge of a particular agenda item, how long the agenda item should last, and what type of meeting control strategy is being employed for that particular agenda item. In the example provided in FIG. 6, the agenda item 84c "Naming Conventions" uses the chairman meeting control strategy. In this strategy, the chairman of the meeting, in this case Gregorio Cruz, can communicate with other meeting members as he so desires. However, if other members wish to communicate, they must first receive permission from the chairman. The conferencing system of the present invention thus provides pop-up menus as a means for talk requests.

Users wishing to communicate with everyone in the meeting can click on the table. A menu will appear with several options including a Talk-Everyone choice. However, users wanting to communicate with a particular member can click on the image of the person. Another menu will appear with more options including a Talk option. As mentioned above, depending on the current meeting control strategy,

The facilitator agent makes use of the information included in the agenda 84 and performs certain functions (e.g. provides prompts, hints, suggestions or the like) in response to the information. For example, since the third agenda item 82c specifies "chairperson," the facilitator agent may prompt/remind the chairman to use a boardroom visual environment (see environment 90 in FIG. 7).

Similarly the facilitator agent may recognize through the preplanned agenda 84a-84c or through recognition of comments made by a participant in the meeting that portions of the meeting may be better suited to a lecture hall environment 92 or a brainstorming mode 94 in which case the facilitator agent would prompt/remind the appropriate participant to use the appropriate ones of the visual environments (e.g. environments 90, 92, 94 in FIGS. 7-7B).

Referring again to FIG. 5A, the facilitator agent 50 builds a rapport with the conferring individual through an interface technique that builds trust between user and agent. Initially the agent is encoded with very basic heuristics and will not make any independent decisions. The agent has a caricature representation 62 that informs the user of its current process state (thinking, suggesting, gratified, disappointed, or confused).

As decision points arise, the agent makes suggestions and through the caricature representation 62 shows expressions of sadness or happiness dependent upon the reaction of the human user. As the agent 50 suggests process interventions, the user may either reject or accept them. Eventually the agent builds thresholds for decisions that may be taken without user intervention. This interactive agent scheme ensures that a trusting relationship is built between agent and user.

FIGS. 7-7B show user interfaces and controls for the collaborative system of the present invention. Environment 90 (FIG. 7) is a boardroom/chairperson environment, environment 92 (FIG. 7A) is a lecture environment and, environment 94 (FIG. 7B) is a freestyle environment.

Each agent has access to information regarding the user as well as the negotiation process. This includes a thresholded encoding of user preferences for agent autonomy (i.e. an indication of what decisions the user is willing to delegate to the agent). The information is supplied to the agent through a set of screens that set user preferences. The current understanding of the agent is displayed to the user who can then set the tolerances for the agents automated activity. The agent decisions on process interventions are based on the following aspects of the agent environment: (a) the current topic's recommendation list; (b) threshold levels indicating user preferences; (c) an averaged record of the participation of each participant in the negotiation process; and (d) the complete conversation model of the ongoing negotiation.

It is important to note that the agent has no direct understanding of the topic being discussed except for that provided by a meeting agenda. Although topic understanding is helpful in assessing the relative importance of certain individuals in the negotiation process, it is prohibitively difficult to realize with current knowledge understanding technology. This difficulty is addressed by specifying issue owners for various stages on the meeting agenda. This mapping should indicate the relative importance of the participant towards the resolution of a specific design conflict.

In one embodiment, the agent behavior is relatively simple. The agent generates a vector representing: (1) the averages of the on-line time each individual has consumed; (2) the amount of recommendations generated regarding a specific intent; (3) the average time a pending speaker waits before he/she is given access to the floor; and (4) the relevance of the topic to each speaker (as indicated in the meeting agenda).

The vector is generated simply by weighting the data elements described and placing the result for each dimension into a vector data structure. Its simply a 4D vector with a weighted element in each dimension generated from the four dimensions described in the preceding paragraph.

At the present time, the facilitator agent only distinguishes between three control strategies: (1) Brainstorming/Free, (2) Lecture, (3) Chairperson. A weight is assigned to each of the components of the vector described above and the elements are summed. The weights are assigned at a threshold level and are refined through user adjustments or through a regressive analysis or through a more complex learning algorithm. These weights are adjustable by the user. The resulting scalar is thresholded at three levels to indicate the appropriate meeting process that the agent should proceed to These thresholds are adjusted by the computer as the agent interacts with the user and are also manually adjustable by the user.

This approach is not optimal at determining appropriate process intervention moments. However, controlled experiments have shown that the agent has made reasonable process intervention suggestions to the users.

Referring now to FIG. 8, A forum 100, here set in a boardroom environment 110 includes a social agent 101 and a facilitator agent 104. As has been previously indicated, each participant has an agent that monitors relevant components of the design environment and suggests appropriate actions to be taken by the participant. With continued interaction, the agent learns how to effectively represent each individual within the design environment. Also, the social agent 101 monitors communications to and from its representee. When the social agent 101 detects one or more communications which it recognizes as corresponding to an emotion of the user, the social agent 101 can suggest to the user an appropriate cue in order to reflect the users emotional state. For example a visual cue may be utilizing an icon of a person with a happy or sad face or by changing a color associated with that user. Thus, the social 101 agent can suggest an image (e.g. an icon) or other cue (e.g. audio) which represents the emotional state of the user (e.g. happy, sad, satisfied, concerned, etc. . . . ).

As the agents become more familiar with their representees they can cooperate with the agents of the other conferees to decide upon the appropriate meeting control scheme. The agents reach consensus on a meeting process intervention through a simple majority rule mechanism.

During a conversation, it is an added benefit to understand the emotions of the other individuals. Knowledge of another person's mood can be an incredible asset during an engagement and can in fact lead to a more effective discourse between those involved. Thus, to allow such expression, an Expressions toolbar 102 including a plurality of caricature expressions 201a-102f is used. The toolbar 102 allows an individual using the conferencing system client to change his/her facial features within the collaboration interface to convey to other clients his/her feelings at a particular instant in time. This type of information can be conveyed in a visual fashion so that users do not have to explicitly state how they are feeling. In this particular example, the toolbar consists of a fixed set of images that are either cartoons or simple drawings.

FIG. 8A presents a snapshot of two participants 108a, 108b who have opened their Expressions toolbar 106a, 106b respectively. Participant 108a has chosen a face with a big smile to exhibit his/her satisfaction. However, this functionality can also be utilized by allowing participants to take various pictures of themselves, which can then be used in place of the caricatures shown in toolbars 106a, 106b. In addition, the system has been programmed to recognize a series of emoticons in the text driver. Therefore, a participant can simply use one of the emoticons in his/her text driver and the conferencing system of the present invention automatically recognizes the text and updates the user's image. The Table below provides a list of the emoticons that are currently recognized by the agent.