Human interface system for communicating networked users

Abstract

A human interface system constructed by connecting systems containing a plurality of computers via a network includes site building means for building a plurality of sites that either retain or manage a plurality pieces of data and a plurality of programs, and access managing means for managing users' access to each site. The access managing means has a communication channel between the users established by accessing the sites via the data and programs.

Claims

What is claimed is:

1. In a human interface system constructed by connecting a plurality of computers via a network, at least one of said plurality of computers comprising:

a plurality of virtual sites, each virtual site including at least one object including people and instruments, each virtual site performing at least one of retaining and managing a plurality of pieces of data and a plurality of programs corresponding to characteristics of each object of said at least one object of said plurality of virtual sites;

access managing means for managing access rights of plural users to control access to each of the plurality of virtual sites, said access managing means including first and second communication channels, wherein said first communication channel is a communication channel between two of said plural users, and wherein said second communication channel is a communication channel between one of said plural users and said at least one object of one of said plurality of virtual sites, each of said first and second communication channels established by accessing said plurality of virtual sites via said plurality of pieces of data and said plurality of programs; and

connecting means for connecting said at least two communication channels to each other,

wherein said at least one of said plurality of computers includes a link information of sites existing on at least another one of said plurality of computers and is capable of establishing communication channels between each of said plurality of virtual sites of said at least one of said plurality of computers and said at least another one of said plurality of computers.

2. A human interface system according to claim 1, wherein each of said plurality of virtual sites comprises:

processing means comprising:

a model section for defining an internal state and operation of said each of said plurality of virtual sites;

a view section in which display information is stored; and

a control section in which processing of an input from one of said plurality of users is written.

3. A human interface system according to claim 2, further comprising display means for displaying a state of an agent.

4. A human interface system according to claim 1, wherein each virtual site of said plurality of virtual sites contains an identifier that distinguishes said each virtual site from each other virtual site of said plurality of virtual sites.

5. A human interface system according to claim 4, further comprising retrieving means for retrieving said identifier for each of said plurality of virtual sites.

6. A human interface system according to claim 5, wherein said retrieving means comprises means for retrieving said identifier for a current site accessed by one of said plurality of users from said plurality of virtual sites.

7. A human interface system according to claim 5, wherein said retrieving means comprises means for retrieving said identifier of one of said plurality of sites under an instruction of one of said plurality of users.

8. A human interface system according to claim 5, wherein said retrieving means contains means for retrieving said identifier from said one of said plurality of virtual sites by using at least one of a telephone number and an address.

9. A human interface system according to claim 1, wherein said access managing means contains:

request analyzing means for recognizing the data entered by the user and analyzing its request; and

response output means for creating a response to the analysis result of said request analyzing means and outputting the response; and

is constructed as a site processing means for enabling a dialogue with said user.

10. A human interface system according to claim 9, wherein said access managing means further contains a knowledge retrieving means for storing knowledge information that said user has and retrieving said knowledge information on the basis of the analysis result at said request analyzing means.

11. A human interface system according to claim 1, wherein said access managing means comprises:

message creating means for creating a message by an addresser;

degree of urgency judging means for judging a degree of urgency concerning a message;

transmitting means for attaching said degree of urgency to said message and transmitting said message to a desired addressee;

state judging means for judging the state of said addressee;

degree of urgency arbitrating means for obtaining a degree of importance of said message based on the degree of urgency transmitted by said transmitting means and the state obtained by said state judging means;

media converting means for determining a method of processing said message based on the degree of importance obtained by said degree of urgency arbitrating means; and

presenting means for presenting said message according to the method of processing said message determined by said media converting means.

12. A human interface system according to claim 1, wherein said access managing means includes:

information broadcasting means for collecting various types of information from said user at each of said sites, processing them, and distributing the results to said user at each of said sites;

information selecting means for selecting information distributed by said information broadcasting means;

presentation mode determining means for determining a presentation mode of the information selected at said information selecting means; and

information presenting means for presenting said information according to the presentation mode determined by said presentation mode determining means.

13. A human interface system according to claim 1, wherein said access managing means includes:

input means for inputting a retrieving instruction to retrieve a desired program;

program retrieving means for retrieving a program directly usable from said computer in response to the retrieving instruction inputted from said input means;

message synthesizing means for converting said retrieving instruction into a transmission message expression to said network when program retrieving at said program retrieving means has failed;

network transmission/reception means for transmitting a transmission message created at said message synthesizing means to the network and receiving a reception message corresponding to said transmission message; and

message analyzing means for relating said reception message from said network transmission/reception means, said transmission message, and said retrieving instruction with each other.

14. A human interface system according to claim 1, wherein said access managing means comprises:

means for determining a theoretical distance between individual users of said plurality of users when said plurality of users attend an electronic conference;

selecting means for selecting either global communication aiming at communication between all of said users attending the electronic conference or local communication aiming at communication between only part of said users attending the electronic conference; and

means for determining the feasibility of local communication on the basis of the theoretical distance between the users available for local communication when local communication is selected at said selecting means .

Description

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a human interface system for use with a plurality of computers connected through a distributed network.

2. Description of the Related Art

A human interface for computer systems (hereinafter, referred to as a human interface system) has been developed mainly with a view to supporting various activities in a single independent computer. Recently, various computer tools have been readily available. They include a tool that places multiple windows on a bit map display and processes multiple jobs in parallel, a tool that allows file creation and deletion by means of a graphical user interface, and a tool that provides data management using a spread sheet.

The popularization of computer networks enables various services including electronic mail through local area networks or wide area networks.

For such human interface systems, various computer tools are modeled according to a concept termed what is called desk top metaphor. Such human interface systems make it easy for the user to access and operate various conventional single-computer tools. Those tools are generally used by the user only after the user has learned their functions and how to operate them. Services handled by the computer are becoming enormous. Actually, the user, carrying out his work or duties, often looks for and refers to data that he does not know where it is, or accesses various computer services which he uses less frequently. Such actions, however, are difficult to carry out in the framework of a conventional human interface.

In such a human interface system, telephone, electronic mail, talk or the like are widely used as systems that allow exchange of messages between terminals connected through a network. In the existing systems, however, unimportant telephone calls sometimes interrupt urgent work or important electronic mails are often left unread for a long time. This is because the system cannot appreciate the working conditions on the called party side as well as the importance of the message the calling party has intended to give and consequently can provide neither the called party nor the calling party with messages with good timing.

The above conventional human interface system cannot transmit messages with appropriate timing according to their importance in such a manner that important messages of great urgency are presented to the called party even if he is at work, whereas messages of less urgency are presented when the called party is not busy.

Similarly, office automation has been popularized remarkably. For example, a work station system has been developed which has a work station with a computer system function installed on each user's desk and allows them to connect to each other and a host computer via a network to communicate with one another.

In such a work station system, when the user wants to get desired information while continuing his work on his work station, he inquires of the user at another work station via the network or calls the host computer and reads the desired information from a data base.

The user at each work station positively inquires of another work station or the host computer in anticipation of getting desired information.

This means that the user cannot get what he wants unless he inquires positively. For this reason, even when acquiring information of less importance, not directly related to the tasks at his work station, the user has to stop what he is doing at his work station temporarily to operate for the acquisition of information, which makes it a troublesome job to get information.

As described above, with the conventional work station system, since the user at each work station cannot get information unless he inquires positively of another work station or the host computer, he has to suspend what he is doing at his work station each time he tries to get information, thus making it a bothersome job to acquire information.

As the processing capability of the computer is expanding, various application programs have been developed for use on such increasingly powerful computers. Such application programs include a text-file tool, a graphic tool, a spreadsheet tool, electronic mail, a schedule management tool, and a database retrieving tool. The computer is not merely a high-speed computing machine but is now serving as a machine which provides an environment that efficiently supports various intellectual activities.

When the user tries to do work, he sometimes spends a lot of time in getting the work done without using the computer or creates a new program even if a function supporting the work is prepared on the computer, because he does not know it. In addition, although the user knows that there is a function he wants on the computer, he cannot use the function after all, because he does not know how to use it.

To solve such problems, there is provided an operating environment where the functions prepared on the computer appear on the screen in the form of menus or icons and are each selected by a pointing device such as a mouse to start each function. With such menu or icon representation, however, there is a limit on what is displayed on the computer screen. Thus, as the number of application programs increases, there may be a case where their menu cannot be displayed on a single screen. To overcome this problem, the menu is made hierarchical or one screen scrolls to another according to the menu.

When the number of application programs has reached approximately 1000, however, the user has to spend a lot of time and labor in searching the menu or icons for the desired function. Therefore, such an arrangement is also difficult to use.

When each type of application program has its own manual, each manual can be stored previously on the computer. In this case, a system can be considered which enables retrieving of a desired function from the manual by specifying a particular Key word, using full-text retrieving techniques.

When the desired application program does not exist on the computer, the manual on the program does not exist as a matter of course, so that the manual cannot be retrieved. Further, even if the desired application program exists, there may be a case where it cannot be retrieved because there is no explanatory text such as a manual on the program. In the case of old computer systems, the user knows what kinds of programs are on the computer only after he asks people around him.

Concerning human interface systems, companies have been introducing electronic conference systems, which allow persons far away from each other to exchange views by means of telephones, television, work stations, etc.

In the electronic conference system, a television camera is prepared for each meeting place where participants have gathered and the scene of the meeting place from each camera is switched so as to create an atmosphere where participants feel as if they were discussing in the same place, by displaying the face of the person now expressing his opinion on the screen, for example.

With such an electronic conference system, however, the meeting proceeds while the participants are looking at a close-up of the person expressing his opinion on the TV screen, which is different in atmosphere from a meeting where the individual participants actually sit across a conference table from each other and exchange their views, with the result that they are liable to have an awkward feeling which would impair a smooth progress of the meeting.

SUMMARY OF THE INVENTION

A first object of the present invention is to provide a human interface system capable of allowing functional activities similar to those in the real world.

A second object of the present invention is to provide a human interface system capable of arbitrating the degree of urgency based on the urgency on the calling party side and the working conditions on the called party side, and controlling message presentation timing and mode in a network system having media including telephones, electronic mails, facsimiles, pagers, and answering machines.

A third object of the present invention is to provide a human interface system which enables the user to get various types of information easily while continuing his work.

A fourth object of the present invention is to provide a human interface system which allows the user to flexibly retrieve an application program containing a desired function.

A fifth object of the present invention is to provide a human interface system which enables a conference to proceed in a manner close to reality and to progress smoothly.

A first human interface system of the present invention constructed by connecting systems containing a plurality of computers via a network is characterized by the computer system comprising: site building means for building a plurality of sites that either retain or manage a plurality pieces of data and a plurality of programs; and access managing means for managing users' access to each site, wherein the access managing means has a communication channel between the users established by accessing the sites via the data and programs.

Further, the first human interface of the invention is characterized in that the access managing means further contains a knowledge retrieving section for storing knowledge information that the user has and retrieving the knowledge information on the basis of the analysis result at the request analyzing section.

With the first human interface system of the present invention, for example, an office is assigned as a network composed of a system having a plurality of computers, and a virtual site is built out of various things (desks, various tools, etc.) in the office. The access managing means enables the user to access the virtual site to get desired information.

By using a site agent as the access managing means for a dialogue with the user and introducing an agent called a person having an automatic response function for access from the user, simple and routine exchanges are automated when a user accesses another user for exchanging information. Only when necessary, the actual user deals with the exchange. Unlike a conventional interface with a single independent program, movement in the computer internal world and a dialogue with another person are allowed. Therefore, when the actual user performs his duties, it is very easy to retrieve and refer to data in a place unknown to the user or use various types of computer services less frequently used.

As described above, with the first human interface system of the present invention, various types of virtual sites can be built on the computers on the network to allow communication between users accessing the virtual sites, which enables movement of virtual sites to make it natural and easy to access information in different places.

A second human interface of the present invention is characterized by comprising: message creating means for creating a message; degree of urgency judging means for judging the degree of urgency concerning the addresser's message; transmitting means for attaching the degree of urgency to the message and transmitting it to a desired addressee; receiving means for receiving the transmitted message attached with the degree of urgency; state judging means for judging the state of the addressee on the basis of the received message; degree of urgency arbitrating means for obtaining the degree of importance of the message on the basis of the degree of urgency transmitted by the transmitting means and the state obtained by the state judging means; media converting means for determining a method of processing the message on the basis of the degree of importance obtained by the degree of urgency arbitrating means; and presenting means for presenting the message according to the method determined by the media converging means.

With the second human interface system of the present invention, instead of presenting a message, a message is stored in the storage means or is sent back to the receiving side according to the decision at the media converting means.

With the second human interface system of the present invention, the degree of urgency of a message created on the transmitting side is automatically judged, the message with this degree of urgency is transmitted, the state of the receiver on the receiving side is automatically judged, the degree of importance of the message is obtained by arbitrating between the transmitted degree of urgency and the state of the receiver, and the message is processed suitably (media conversion, presentation, return, etc.) on the basis of the degree of importance. Therefore, message communication can be achieved with the optimum timing and state obtained by arbitrating between contradicting requirements on the transmitting and receiving sides. That is, smooth message communication can be performed with the degree of urgency desired by the transmitting side well compromising with the working efficiency on the receiving side, which provides both the transmitting and receiving sides with an easy-to-use human interface system.

A third human interface system of the present invention where a plurality of work stations are connected to a network to which an information broadcasting station is connected, is characterized by the information broadcasting station comprising: information collecting means for collecting various types of information from the work stations; information processing means for processing the collected information; and information broadcasting means for distributing the processed information to the work stations; and each of the work stations comprising: selecting means for selecting from the information distributed from the information broadcasting station; presentation mode determining means for determining a presentation made of the information selected at the selection means; and presenting means for presenting the information according to the presentation mode determined at the presentation mode determining means.

With the third human interface system of the present invention, the information broadcasting station connected to the network to which work stations are also connected, broadcasts various types of information collected from different places, including each station, to the individual work stations. Thus, the user at each work station can create an environment where he can readily get various types of information from the screen display whose presentation mode is well designed, without suspending his duties.

The information broadcasting station positively collects insignificant information other than the information each user directly uses in his duties, processes it into a form easily understandable to the user, and broadcast it. At each user's terminal, such insignificant information is displayed to the user's liking in a manner that does not interfere with his work. Therefore, the user, while doing his work, can know the circumstances (including the state of the other users connected to the network) and the surrounding atmosphere with ease, which were not available on conventional network terminals.

For example, even in a satellite office, it is possible to create an atmosphere of a common usual office, using the system of the invention.

A fourth human interface system of the present invention constructed by connecting systems containing a plurality of computers via a network, is characterized by comprising: an input section for inputting a retrieving instruction to retrieve a desired program; a program retrieving section for retrieving a program directly usable from the computer in response to the retrieving instruction inputted from the input section; a message composing section for converting the retrieving instruction into a transmission message expression to the network when program retrieving at the program retrieving section has failed; and a network transmission/reception section for transmitting the transmission message created at the message composing section to the network and receiving a reception message corresponding to the transmission message.

The program retrieving section retrieves programs stored in the program storage section on the basis of documents such as manuals on how to use programs.

Further, when the program retrieving section has failed to retrieve a program stored in the program storage section, the message composing section converts the user's request into a message to inquire of another user. For example, a message is converted into an electronic mail or a piece of electronic news.

The network transmission/reception section transmits a message sent from the message composing section to the communication network. The transmitted message is sent to a user different from the user retrieving the program.

The other user, receiving the message, may know the place where the application program the original user wants (the name of a directory in a case where program files are managed in a hierarchical directory like UNIX) is or how to use. In this case, the user replies to the received message.

When the other user send a reply, the message analyzing section receives the message via the network transmission/reception section. By analyzing the message, it is judged whether or not the message is a reply to the original message.

When a program is contained, it is reported to the program retrieving section and the program is stored in the program storage section.

With the fourth human interface system of the present invention, the user can inquire about a desired application program of the computer by keyword or the like. Further, when the program to be retrieved cannot be found, the retrieving request is converted into a message to inquire of another user via the communication network. Thus, when the other user knows such a program and can answer the message, he can send a reply to the original user. Therefore, even if the user cannot get a desired program on his computer, he can easily get the desired program.

As explained above, with the fourth human interface system of the invention, the user can inquire about a desired function of the computer by attempting to retrieve information. When the retrieving has failed, the process of inquiring of another user can be performed automatically on the computer. Thus, in a computer environment where more than one user uses a plurality of computers, it is possible to make good use of program resources.

A fifth human interface system of the present invention constructed by connecting a plurality of computers serving as electronic conference terminals via a network, is characterized by comprising: distance determining means for determining the theoretical distance between the individual electronic conference terminals; display means for determining the seating order of users at the electronic conference terminals on the basis of the theoretical distance determined at the distance determining means and displaying the users at the electronic conference terminals according to the seating order of the users; selecting means for selecting either global communication aimed at communication between all of the electronic conference terminals connected to the network or local communication aimed at communication between only part of the electronic conference terminals; and feasibility determining means for determining the feasibility of local communication on the basis of the theoretical distance between electronic conference terminals available for the local communication, when local communication has been selected at the selecting means.

With the fifth human interface system of the present invention, the seating order of the individual electronic conference terminal users can be determined, according to the theoretical distance between electronic conference terminals, to display each electronic conference terminal user on the screen, following the seating order, and the feasibility of local communication between part of the electronic conference terminals can be decided based on the theoretical distance between the electronic conference terminals concerned. Thus, it is possible to create an atmosphere similar to that of actual conferences where the attendants are sitting around a table, using screen displays of the individual electronic conference terminal users. Further, since only electronic conference terminal users topographically close to each other are permitted to perform local communication, this makes it possible to progress the conference under conditions similar to reality, which frees the users from a sense of incongruity, thereby enabling smooth progress of the conference.

Additional objects and advantages of the present invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present invention. The objects and advantages of the present invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate presently preferred embodiments of the present invention and, together with the general description given above and the detailed description of the preferred embodiments given below, serve to explain the principles of the present invention in which:

FIG. 1 is a block diagram showing the computer network structure of a human interface system according to a first embodiment of the present invention;

FIG. 2 is a schematic diagram of a computer according to the first embodiment of the present invention;

FIG. 3 is a diagram showing a basic construction of site server and a client;

FIG. 4 is a conceptual diagram visualizing the structure of a virtual site;

FIG. 5 is a diagram showing the structure of each agent in site DB;

FIG. 6 is a view of a file cabinet and a desk;

FIG. 7 is a view of a display image at a virtual site;

FIG. 8 is a diagram showing a structure of connection between virtual sites;

FIGS. 9A and 9B are flowcharts for person transfer processing;

FIG. 10 is a view of a display specifying where to go;

FIG. 11 is a flowchart for site list element processing at step SA3 of FIG. 9A;

FIG. 12 is a flowchart for person's name list processing at step SA5 of FIG. 9A;

FIGS. 13A through 13D are flowcharts for postal-code number/address processing;

FIG. 14 is a flowchart for transfer site specifying by language input;

FIG. 15 is a view of an area site screen;

FIG. 16 is a view of an access screen to a person;

FIG. 17 is an explanatory diagram for establishment of communication channel between persons;

FIG. 18 is a view of a screen display for communication between persons;

FIG. 19 is a block diagram related to a person's response action;

FIG. 20 is a block diagram of the automatic processing section;

FIGS. 21A through 21C are flowcharts for the processing at the request analyzing section;

FIGS. 22A and 22B are flowcharts for the processing at the problem solving section;

FIGS. 23A through 23E are flowcharts for the processing at the response judging section;

FIG. 24 is a schematic diagram of a human interface system according to a second embodiment of the present invention;

FIGS. 25A to 25C are flowcharts for the processing at the calling party's degree of urgency judging unit;

FIG. 26A is a block diagram showing the construction of the calling party's degree of urgency judging unit;

FIG. 26B is a view of an example of various degree of urgency added by the calling party's degree of urgency judging unit;

FIG. 27 is a flowchart for the processing at the sender;

FIG. 28 is a block diagram of the sender;

FIG. 29 is a flowchart for the processing at the called party's state judging unit;

FIG. 30 is a flowchart for the processing at the degree of urgency arbitrating unit;

FIG. 31A is a block diagram of the degree of urgency arbitrating unit;

FIG. 31B is a table listing weight on each axis of degree of urgency used in the processing at the degree of urgency arbitrating unit;

FIG. 32 is a flowchart for the processing at the media converting unit;

FIG. 33 is a block diagram of the media converting unit;

FIGS. 34A through 34C are views of operating screens in the system of the second embodiment;

FIG. 35 is a view of a screen requesting the confirmation of added degree of urgency;

FIG. 36 is a schematic diagram of an arrangement for correcting the added degree of importance on the calling party side;

FIG. 37 is a schematic diagram of an arrangement for correcting the added degree of importance on the called party side;

FIG. 38 is a schematic diagram of an arrangement for letting the calling party know how the presented message has been handled;

FIG. 39 is a schematic diagram of an arrangement for storing another piece of information in the response recording section of FIG. 38;

FIG. 40 is a schematic diagram of an arrangement for allowing the calling party to learn the relationship between the degree of urgency and how to handle the message;

FIG. 41 is a schematic diagram of the state judging unit provided with another function;

FIG. 42 is a flowchart for the addition of the degree of urgency of message to an unspecified called party;

FIG. 43 is a schematic diagram of an arrangement where the calling party degree of urgency judging unit is provided on the called party side;

FIG. 44 is a schematic diagram of a human interface system according to a third embodiment of the present invention;

FIG. 45 shows various tasks performed at the work station;

FIG. 46 is a schematic diagram of the work station;

FIG. 47 is an explanatory diagram of the operation of the information broadcast receiving section;

FIG. 48 is an explanatory diagram of the operation of the information presenting section;

FIG. 49 is a schematic diagram of the information broadcasting station;

FIG. 50 is a schematic diagram of the control section;

FIG. 51 is an explanatory diagram of the operation of the information acquiring section;

FIG. 52 is an explanatory diagram of the operation of the information summarizing section;

FIG. 53 is an explanatory diagram of the operation of the information broadcasting section;

FIG. 54 is a view for explaining a screen display in the third embodiment;

FIG. 55 is a view for explaining a screen display in the third embodiment;

FIG. 56 is a view for explaining a screen display in the third embodiment;

FIG. 57 is a view for explaining a screen display in the third embodiment;

FIG. 58 is a view for explaining a screen display in the third embodiment;

FIG. 59 is a schematic diagram of a human interface system according to a fourth embodiment of the present invention;

FIG. 60 is a block diagram of the input section and the controller;

FIG. 61 is a flowchart for the processing at the program retrieving section;

FIG. 62 shows part of a manual;

FIG. 63 is a block diagram of the program retrieving section;

FIG. 64 is a flowchart for the processing at the retrieving hysteresis judging section;

FIGS. 65, 66A and 66B are diagrams showing part of the retrieving hysteresis storage section;

FIG. 67 is a flowchart for the processing at the message synthesis section;

FIG. 68 shows an example of a retrieving keyword;

FIG. 69 shows an example of an interrogative sentence;

FIG. 70 is a diagram showing an example of part of the contents of the transmission storage section;

FIG. 71 is a block diagram of the message synthesizing section;

FIGS. 72A and 72B are diagrams showing an example of the contents of the addressee information storage section;

FIG. 73 is a flowchart for the processing at the message synthesizing section;

FIG. 74 shows a synthesized message;

FIG. 75 shows an example of a responsive sentence;

FIG. 76 is a block diagram of the message analyzing section;

FIG. 77 is a flowchart for the processing at the message analyzing section;

FIG. 78 is a block diagram of the storage section;

FIG. 79 is a diagram showing an example of data storage in the program storage section;

FIG. 80 is a schematic diagram of a human interface system according to a fifth embodiment of the present invention;

FIG. 81 shows a database for participants used in the fifth embodiment;

FIG. 82 is a schematic diagram of electronic conference terminals used in the fifth embodiment;

FIG. 83 shows the format of a packet used in the fifth embodiment;

FIG. 84 is a flowchart for the operation of the fifth embodiment;

FIG. 85 is a flowchart for the operation of the fifth embodiment;

FIG. 86 is a flowchart for the operation of the fifth embodiment;

FIG. 87 is a flowchart for the operation of the fifth embodiment;

FIG. 88 illustrates an example of a screen display at an electronic conference terminal in the fifth embodiment;

FIG. 89 illustrates an example of a screen display at an electronic conference terminal in the fifth embodiment;

FIG. 90 illustrates an example of a screen display at an electronic conference terminal in the fifth embodiment;

FIG. 91 illustrates an example of a screen display at an electronic conference terminal in the fifth embodiment;

FIG. 92 illustrates an example of a screen display at an electronic conference terminal in the fifth embodiment;

FIG. 93 illustrates an example of a screen display at an electronic conference terminal in the fifth embodiment;

FIG. 94 illustrates an example of a screen display at an electronic conference terminal in the fifth embodiment; and

FIG. 95 is a schematic diagram of a modification of the fifth embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring to the accompanying drawings, embodiments of the present invention will be explained.

FIG. 1 shows a computer network structure of a human interface system according to a first embodiment of the present invention. In the computer network of FIG. 1, computers 100a through 100d, such as work stations or personal computers, are connected to each other through local networks 900a and 900c and a wide network 900b or the like. These networks 900a through 900c enable the individual computers 100a through 100d to exchange data with each other.

FIG. 2 is a concrete block diagram of a computer in the first embodiment.

In FIG. 2, a display section 110 is a display unit or the like. An input section 120 is an input unit, such as a keyboard and a mouse from which the user enters characters or commands, a panel screen from which the user enters pointing information on position. A communication section 130 inputs and outputs data from and to another network. A storage section 140 is a storage unit such as a disk. An agent processing section 150 supports agent programming. A control section 160 allows the display section 110 to control the agent processing section 150.

An agent program will be explained.

An agent defining section 170 and a stage 180 execute an agent program.

The agent defining section 170, which is provided in the storage section 140, is capable of defining various agents (programs). An agent is composed of a model section, a view section, and a control section, as normally defined. The model section defines the internal state and operation of the agent. The view section stores information on the display of the agent. The control section contains the descriptions of the processing corresponding to operation of a mouse serving as an input device.

The agent, receiving data called a message from the outside, carries out a process according to the data received. The agent, if necessary, sends messages to other agents. When an agent start message is sent to the agent processing section 150 via the input section 120 or the communication section 130, the agent processing section 150 starts an agent. Starting agent means creating an instance at stage 180 or its executing place, based on the definition of agent.

In describing the agent, it is possible to use classes introduced in normal object-oriented languages, their methods, and successional data. The user can send a message to an agent via the input section 120 (such as a keyboard or a mouse). The agent can present information to the user via the display section 110. The agent can perform execution at stage 180 not only on a computer having the definition of the agent, but also on any other computer. In this case, data transmission and reception is carried out via the communication section 130. It is possible to transfer or copy the agent from one stage 180 to another stage 180.

FIG. 3 shows a basic construction of a site server 200 and a client 300 in the first embodiment of the present invention.

In this configuration, a site server 200 contains a site DB 210, a site DB managing section 220, a visualizer 230, a site handler 240, a display data section 250, and a site agent defining section 260. The client 300 contains a client handler 310, a display processing section 320, an input control section 330, and a client agent defining section 340.

The site DB 210 stores messages sent from various agents.

The site DB managing section 220 manages virtual sites. Specifically, the site DB managing section 220 performs processes including the creating, updating, and deleting of agents in the site DB 210, as well as stores the state of the site DB 210 in the storage section 140.

The visualizer 230 reads out display information on each agent from the site DB managing section 21 and builds screen display data.

The site handler 240 reads out a definition of the virtual site specified by the user from the site agent defining section 260 and performs processes including transmitting a message to create an agent to be put in a virtual site to the site DB managing section 220.

The display data section 250 displays the screen display data built at the visualizer 230.

The site agent defining section 260 stores a definition of the virtual site specified by the user.

The client handler 310, receiving the user's view-specifying information via the input processing section 330, performs processes including requesting the display area information in the view-specifying information of the visualizer 230.

The component elements 210 through 260 of the site server 200 and the component elements 310 through 340 of the client 300 can be basically defined as agents. These components' parts may be placed in either the client 300 or the site server 200. For example, the client handler 310 may be placed in the site server 200. When the processing load at the client 300 exceeds a specified value owing to the load condition and hardware limitations (including storage size and CPU speed) on client 300, the arrangement of the agent may be modified dynamically. Message exchange is achieved via a network between agents operating at the stage 180 in client 300 and the stage 180 in site server 200.

Each component part and its operation will be explained.

FIG. 4 is a conceptual diagram visualizing the structure of a virtual site used in the first embodiment.

In FIG. 4, an entire site corresponds to a department. The site has three rooms in it: the 1st section 410, the 2nd section 420, and the manager's office 430. The 1st section 410 is furnished with a desk 411, a file cabinet 412, and a clock 413; the 2nd section is furnished with a desk 421 and a file cabinet 422; and the manager's office 430 is furnished with a tool box 431, a form box 432, and a job agent box 433. The tool box 431 has various tools in it. In the site, there are also an agent 415 called a person corresponding to a man and a site agent (not shown) deals with questions as to the entire site. In practice, these individual elements are defined in site DB 210.

FIG. 5 shows the structure of each agent in site DB 210 in the present embodiment.

In FIG. 5, an element indicated by .largecircle. corresponds to an agent. The name in .left brkt-top. .right brkt-bot. assigned to .largecircle. is the name of the class indicating the agent. A name directly written is the name of the instance of the agent. Arcs connecting between agents show how agents are connected to one another. An arc means that the following facts are retained in the form of data: the fact that the agent at the leading end of the arc is entered in the list of agents to whom messages can be sent, and the fact that there is the relationship indicated by the name of the ark between two agents. "HAS" and "IN" arcs represent a space positional inclusion relationship between virtual sites in the agent: the agent at the trailing end of the arc specially includes the agent at the leading end of the ark. "HAS" basically indicates an agent statistically positioned and "IN" denotes an agent dynamically positioned.

SITE SPACE AGENT is an agent having the following internal state, method, and view:

Internal state: a list of names and agents (dynamically related agents, agents in the site, and other agents)

Method: display, entry, deletion, modification, provision of information

View: view 1 (two-dimensional model), view 2 (three-dimensional model), view 3 (image)

A room agent is an agent having the following internal state, method, and view:

Internal state: a list of names and agents (agents in the room and the other agents)

Method: display, entry, deletion, modification, provision of information

View: view 1 (two-dimensional model), view 2 (three-dimensional model), view 3 (image)

The name of an agent is stored in NAME of the internal state, and IDs of agents having the above inclusion relationship (by which, the instances of the individual agents can be identified uniquely) are stored in the agent list. In the method field, the operation on the agent and definitions (corresponding to programs) of processing operations, including DISPLAY, ENTRY, and DELETE, are written. In the VIEW field, display information items are written to cope with display modes; there are three types available here: two-dimensional display (plan-view representation), three-dimensional display (elevational-view representation), and image display. Definitions of the other agents will be explained later as the occasion demands. A virtual site as shown in FIG. 5 is managed by the site DB managing section 220.

These processes are started by a message sent from site handler 240 or messages sent from various agents in the site DB 210. The site DB managing section 220 itself is generated by the site handler 240. (This is similar to the start of a normal agent.)

The generation of an agent is carried out as follows.

The name of a class or the defined form of an agent, the settings of the internal state, the ID of an agent in the site where the agent is to be located, and a message to generate an agent of generation position information, are sent from the site handler 240 to the site DB managing section 220. The site DB managing section 220 requests a processing system (not shown) to generate an instance in the specified class in the process of generating instances. The processing system generates an instance at stage 180 and notifies site DB managing section 220 of the generated instance. The site DB managing section 220 produces a channel between itself and the generated instance. Further, the instant ID and the generation position information generated together with a "locate" message are sent to an agent having the ID of an agent in a site where the agent is to be located. An agent who has received a message from site DB managing section 220, produces a message channel between itself and the site DB managing section 220.

The deleting of an agent is achieved as follows.

When a "delete" message is sent to an agent to be deleted, the agent sends a "delete-ready?" message to all message channels. Each agent which has received the "delete-ready?" message, performs the necessary processes, and if necessary, carries out the specified processing of the agent to be deleted. The agent to be deleted waits until "delete-accept" messages from all corresponding agents have arrived. When all the messages have arrived, the agent sends a "IAmDeleted" message to all message channels and issues an instruction to delete itself to the processing system (not sown). This deletes the agent from stage 180.

Updating agents is done by sending messages to agents basically the same manner as the delete process.

A method of displaying the contents of site DB 210 on the display unit at client 300 will be explained. The display process is carried out by the visualizer 230, display data section 250, client handler 310, and display processing section 320 cooperating with each other. The display process is performed at the time when a virtual site is generated (in the initializing stage) or when the display has to be modified owing to a change in the state of site DB 210.

In the initializing stage, the user specifies the generation of the site handler 240, which then creates the site DB managing section 220. The site handler 240 takes out definition of the virtual site specified by the user from the site agent defining section 260, sends a message to create agents to be put sequentially in the virtual site, and creates each agent in site DB 210. After all the initial agents have been created, the site DB managing section 220 creates the visualizer 230 (agent). The visualizer 230 takes out display information on each agent and constructs screen display data in the display data section 250. In the first embodiment, a plurality of display data areas are formed so as to permit a single site DB 210 to have more than one display mode.

Taking a case of representation in a two-dimensional plane as an example, the operations of various elements will be explained.

Part of view information on an agent in site DB 210 (written in the view area in the agent) will be shown below.

"Site Space": 10 {(Name=the first department Two-dimensional view:

Shape: rectangle (height=80, length=130, boundary=(solid line, 3, black), inside=(mesh 1, gray

Three-dimensional view: . . . }

"Room": 11 {(Name=the first section Two-dimensional view:

Shape: rectangle (height=80, length=40, boundary=(dotted line 1, 1, black), inside=(transparent relative position=`0, 40`

Three-dimensional view: . . . }

"File Cabinet": 14 {(Name=the first section file cabinet

Two-dimensional view:

Shape: `File cabinet image` ("the first section") Relative position=(0, 20)

Three-dimensional view: . . . }

"Desk": 15 {(Name=Yamada desk Two-dimensional view:

Shape: desk visual image ("Yamada")

Relative position=(30, 15)

Three-dimensional view: . . . }

In the "Site Space" representing the entire site, a rectangle is specified for the shape of a two-dimensional view. For the parameters of the rectangle, the height, the length, the rectangle's boundary line (type, width, and color of the line), and the rectangle inside (a painted pattern, color) are defined. In the two-dimensional view in "Room," a relative position is defined in addition to the shape definition. This information determines where the Room is placed in the two-dimensional shape in the Site Space directly containing the area of the "Room." For the shape of the cabinet, file `cabinet image` ("The First Section") is specified. This, not shown here, indicates image data whose shape is such that the characters "The First Section" is overwritten at the center of the image data shaped like a file cabinet pattern 10 high and 20 long. Similarly, the second view in "Desk" also represents image data.

FIG. 6 shows an example of a file cabinet and a desk.

The visualizer 230, after an agent has been generated, sends a two-dimensional message "Send2DViews(0)" to the site space agent. The (0) in the message is a value called a display level and is used in processing the hierarchical relations in display. Receiving the message, each agent in site DB 210 sends its ID, the received display level, its two-dimensional view to the calling party of the message. At the same time, each agent sends a message "Send2DViews" (the received display level +1) to the agents to which its message channels are connected under the "has" relationship, and then sends its return message (ID, display level, two-dimensional view) to the calling party of the message. Next, the agents to which the message channel connected under the "in" relationship are processed in a similar manner.

By doing this, the visualizer 230 collects the two-dimensional view information on all the agents in site DB 210 and their display information. These pieces of two-dimensional information are retained as information on the inside of the visualizer 230.

There are roughly two methods of displaying the site state on the screen in client 300 on the basis of the above information. One method is to cause client 300 to take in these pieces of two-dimensional view information and to display them on the actual screen there. The other method is to cause the visualizer 230 to create the image data for screen display in the display data section 250 and send the image information to the display processing section 320 of the client 300 via client handler 310. With the former method, it is possible to select a display mode according to the user's request at client 300, or to set a display level independently at client 300. The latter method has the advantage that the load at client 300 becomes lighter because one display process is carried out on the side of visualizer 230.

An example of image display on the side of visualizer 230 will be explained.

The two-dimensional view information collected by visualizer 230 takes the form of display images of agents overlapping with each other. The visualizer 230 reserves a display area for retaining the display images of sites in the display data section 250. This area, which is similar to an ordinary screen display buffer, is shared by more than one client 300 via a network. The visualizer 230 checks up on the collected two-dimensional view information and writes it in the display area, starting at a piece of two-dimensional information of lower display level. By doing this, the image information at site DB 210 in the above example is set in the display area (shown in FIG. 7).

The operation of displaying the image information set in the display area on the screen at client 300 will be explained. The processes related to client 300 are basically carried out by the handler agent at client 300.

The handler agent at client 300, which is generated by site handler 240, reconizes the ID of the visualizer agent from site handler 240 and reserves a message channel to visualizer 230. The client handler 310 receives the user's view specifying information (a two-dimensional view, in this case) via the input processing section 330, and requests of visualizer 230 information on the display area which contains the view. This request is sent to visualizer 230, which returns the address of the display area for two-dimensional view and the size information to client handler 310. The client handler 310 sets an area value covering the entire display area in a display area defining variable indicating which part of the display area is to be displayed, and then makes a display request to the display processing section 320. The display processing section 320 displays on the display unit the display image of the range indicated by the display area defining numbers at client handler 310 in the display area corresponding to the two-dimensional view.

The pointing by the user and the selection of various commands will be explained.

The pointing by the user is achieved by a pointing device such as a mouse in a manner that moving the mouse cursor. When the user operates the mouse, such as presses the mouse button, at a particular position on the display at the virtual site, the input control section 330 senses the coordinates of the position and informs client handler 310 of the user's event (for example, the mouse button being pressed). The client handler 310 calculates the coordinates of the display data section 250 from the value of the display area defining area and the values of the mouse coordinates, and sends the resulting values together with a "GetAgents" message to visualizer 230. The visualizer 230 obtains agents at the coordinates, and sends their IDs and their display level information to client handler 310. The client handler 310 selects one agent from these agents (normally, an agent of the highest display level), and makes a request via site handler 240 to inform the agent of the user's event, whereby the message is reported to the desired agent.

By such processes, the event entered by the user at the input section 120 is transmitted to the agent in site DB 210. When more than one agent exists at the position specified by the user, client handler 310 may deal with their competition. For example, in a case where a telephone is on a desk and their display level is the same, when the user specifies the position of the telephone and presses the mouse button, the client handler 310 first creates a display for selection of one of the two on the screen and the user can specify the specified agent by moving the mouse to the two display areas.

The movement in connection with the display of an agent will be explained.

The display position of an agent is retained in the form of internal information on agents in site DB 210. The user can communicate with the agent in the method described above and send positional moving commands to the agent. When the positional information on the agent is updated, each agent issues a display modify request to visualizer 230. The visualizer 230 moves the agents that have made moving requests, computes the display areas for the agents that require rewriting on the screen as a result of the movement, and rewrites a display portion of the display buffer.

The person agent will be explained.

Person agents, which are in the virtual site, are characterized by having the mapping relationship with actual human beings. Further, the person agent is a type of agent and has the following information as internal information:

At least one home site ID and one home agent ID

A current site ID and positional information in the current site

Various pieces of personal information (name, post, address, phone number, user's identifier)

View information

Status information

Site hysteresis information

The home site is the ID of the initial value site of the person and can have more than one site ID. The home agent ID is an agent that the person has. Normally, a desk agent corresponds to a home agent ID. In this embodiment, the home agent is called a "personal desk." The personal desk not only serves as a person's working environment, but also retains a method of accessing a person. In this embodiment, the method of accessing a person is expressed by the current site ID and the person agent ID themselves. The current side ID is the ID of the virtual site where the person has visited at that time, The positional information at the current site is information indicating the position of the agent in the site explained above. The various pieces of personal information include various pieces of information corresponding to the person, including the name, the post, the address, and the phone number. The view information is information on the view of the agent. The status information includes various pieces of information indicating the states of the person, which will be explained later. The personal desk holds the person's ID, the person's current site ID, the person's status information, the view information, and links to various working agents.

The procedure followed by the user in starting the computer will be explained in connection with the virtual site and the person.

When no computer is used, the person exists in site DB 210 in the home site, and its status is "not-login." When the user has followed the normal computer starting procedure, various common processes, including the input, display, and communication processes, are generated. After this, a generation request of client handler 310, together with the user identifier, is sent to site handler 240. The site handler 240, using the user identifier as a key, retrieves a person agent in site DB 210 and informs the client handler 310 of the ID of the person agent. The client 300 server performs the process of displaying the state of the site on the screen. The client handler 310 sends a message to change the status of the person agent to "login" to the person agent via site DB 210. The person agent, receiving the message, sends a display update message to the visualizer 230 in order to change its view display mode from "not-login" to "login."

By the above operation, the display state of the person agent at the display section is changed, whereby the other users understand that the user corresponding to this person has begun using the computer.

The movement of the person between virtual sites will be explained. Moving from a virtual site A to another virtual site B is achieved using the ID of virtual site B at the destination.

The operation of the above two virtual sites will be explained. Site A at the starting point, receiving a site movement request for a person (including the movement site ID), makes person reference to the site server 200 with the ID. The person reference is the process of requesting person information necessary to check for movement of the server at site B, getting the requested information from the person, sending it to site server 200 at site B, and inquiring whether movement is possible or not. When person reference is successful, the starting-point server transfers the person agent to the destination server and deletes the person agent from its own site. The destination site server 200 enters the received person agent in its own site DB 210.

For the person to move, the virtual site's ID is necessary. Methods of obtaining the virtual site's ID will be explained.

A first method is to get information from the current site where the person is.

As shown in FIG. 8, a company organization or the like is defined as a plurality of virtual sites connected in a network. For arcs in the network, two virtual site-related names can be defined. For example, in FIG. 8, COMPANY, GUARD, RECEPTION, DEPARTMENT 1, DEPARTMENT 2, and LIBRARY are connected under the relationship of "is-section-of", which is the inclusion relationship in the organization. DEPARTMENT 1, DEPARTMENT 2, LIBRARY, and RECEPTION are connected under the relationship "is-neighbor-of." The connection-related names, the other party's site names, and IDs of them are stored as the other site link information in the model section of each virtual site agent. This information is called site link information, and sites connected to each other are called adjacent sites. This site link information enables the person to take out information on the virtual sites connected to the current site.

A second method is to allow the person itself to store information on the virtual site.

Under instructions from the user, the name of the virtual site and its ID can be stored in the person and the information (called personal site information) be retrieved as required. By taking in usually coming and going sites into the person in the form of information, operations such as following more than one virtual site can be simplified. By storing a person's name and his home agent's ID (called personal person information), not the virtual site, the site where an individual is can be retrieved. Further, the person holds information called site hysteresis information. The site hysteresis information is a combination of the virtual site's name and ID and contains the hysteresis of the sites (except for the home site) which the person has visited.

A third method of identifying sites according to a phone number.

The user enters a specific phone number, which is used as a key to retrieve the site ID. A phone number agent is prepared which has a function to take out the site's ID from the phone number. The phone number agent has a table with the ID of the virtual site corresponding to the phone number and returns the ID in response to requests from various agents. For the inquiry phone number as long as the area code, the ID of a virtual site representing the area (called an area site) is returned. The area site inquires about the area and acts as an intermediary in moving to virtual sites. When phone numbers inquired do not exist, the phone number agent returns the fact that the desired numbers do not exist and the area site's ID.

A fourth method is to identify sites according to a postal code number and address.

The user enters a specific postal code number and a specific address, which are used as keys to sense the site ID. With this method, a postal code number and address agent similar to a phone number agent is prepared to enable retrieving of the site ID on inquiry.

When there is no site corresponding to the address specified, the fact and the ID of a higher-level area site are returned.

A fifth method is to identify sites by inquiry using natural language.

An agent is prepared which allows the user to enter a desired place using natural language, which is then analyzed to identify the site's IDs (site uniforming agent). The site uniforming agent has the function of sensing an appropriate site ID from the character string entered by the user.

A sixth method is to move to a site in the called party in communication between persons. It is possible to get the site ID by inquiring about the current site of the other party's person.

Referring to FIGS. 9A through 14, the process of moving a person from one site to another in the above six items will be explained.

The user carries out command selection at the input section 120 to select a MOVE command. This request is transferred from the person agent to the move processing agent. The move processing agent performs processes according to the flows shown in FIGS. 9A and 9B. The move processing agent takes out the personal site information, the personal person information, the site hysteresis information, and the current site information which the person requested to move has.

The move processing agent displays a "where-to-go setting window as shown in FIG. 10 on the user screen, and at the same time, sets the "destination site ID" to NULL, the destination site ID retaining a site ID where the person will go. The name of the current site appears at the position "PRESENT" on the display screen, and the name of a place where the person will go appears at "DESTINATION." On the site list, site names obtained from the personal site information and adjacent site names obtained from the site link information are displayed. In this case, when the person's current site is a personal site of the person, the personal site is given priority in display, for example, it is placed at the head. Otherwise, adjacent sites are given priority in display. When the current site is not the home site, however, the home site is given priority in display. At the position of the name list, names obtained from the personal information are displayed. In the "SITE HYSTERESIS," site's names are taken out from the site hysteresis information and displayed. "WHERE TO GO", "T", "TEL" are fields in which keys for retrieving the destination are to be entered. "STATE DISPLAY" is a field for displaying various messages. The move processing agent then waits for the user to enter and performs a process according to the input.

At step SA3 in FIG. 9A, it is judged whether or not the user's request for a site list has been selected with means such as a mouse. If the site list is requested at step SA3, the move processing agent performs a site list element process (step SA4). FIG. 11 is a detailed flowchart for the site list element process. In the site list element process, the specified site is determined (step SB1), and the site ID is taken out and set in "DESTINATION SITE" (step SB2).

At step SA5 in FIG. 9A, it is judged whether or not the request for the name list has been selected with means such as a mouse. At step SA5, when the name list has been requested, the name list element process is carried out (step SA6). FIG. 12 is a detailed flowchart for the name list process. The home agent of the person corresponding to the specified person's name is taken out (step SC1). The information on the state of the other party's person is requested of the taken-out home agent (step SC2). The state information includes the current site information, home site information, and status (not-login, talking, busy, acceptable, free) of the other party's person. "not-login" indicates a state where the other party cannot access via the computer. "talking" indicates a state where the other party is now communicating with someone. "busy" indicates a state where communication is impossible because of busyness. "acceptable" indicates a state where direct communication should be avoided if possible. "free" means that direct communication is possible at present. The move processing agent takes out the other party's home site ID and sets the ID in "DESTINATION SITE" (steps SC3 and SC4).

At step SA7 in FIG. 9A, it is judged whether or not the user has selected an element of the site hysteresis with means such as a mouse. At step SA7, if the hysteresis site has been requested, the hysteresis site element process is performed (step SA8). At step SA8, as with the site list element process, the site ID is taken out and set in "DESTINATION SITE."

At step SA9 of FIG. 9A, it is judged whether or not the user has specified the "T" field or the "ADDRESS" field and entered data in these fields with a means such as a keyboard. At step SA9, if the "T" or the "ADDRESS" field is filled, the following postal code number and address process is carried out (step SA10). Otherwise, control proceeds to step SA11 in FIG. 9B.

FIGS. 13A through 13D are detailed flowcharts for the postal code number and address process.

By sensing the decide key on the keyboard or the execute key on the display screen, it is sensed that the user has finished entering data (step SD1). it is judged whether the user has entered data in the "T" field (step SD2). At step SD2, if the user has entered data in the "T" field, the site handler 27 retrieves the site ID using the postal code number as a key (step SD3). The operation at step SD3 is performed by receiving the postal code number and sending a message to a special agent that only returns a corresponding site ID and the address information. The agent holds a table in which more than one combination of a postal code number, a site ID, and address information is written. The site handler 27 retrieves this table. When the site handler 27 has succeeded in retrieving the table, it sends the site ID and the address information to the site DB managing section 220. When it has failed to retrieve the table, it sends a failure message to the site DB managing section 220.

It is judged whether retrieving of the postal code number is successful or not (step SD4). When the retrieving has failed, the effect that the retrieving has failed is displayed in the message field and the process is terminated (step SD5 in FIG. 13D). At step SD4, when the retrieving of the postal code number is successful, the obtained address information is displayed in the address field on the display screen (step SD6 in FIG. 13A), the obtained site ID is set in the destination site, and the process is terminated (step SD7). Although not described in the above process, the name of the site ID is retrieved (an inquiry is made of the site) and the name is displayed in the destination field on the display screen.

At step SD2, when the user has not entered data in the "T" field, it means that he has entered data in the address field, so that retrieving is achieved with address as a key (step SD8). This retrieving process is performed by sending a message to a special agent that receives the address character string and the positional information on the current site (optional) and returns a corresponding site ID and an address character string. The agent divides the received address character string into specified units, using an address dictionary, determines elements including the administrative divisions of Japan: the prefectures plus Tokyo, Hokkaido, Osaka, and Kyoto, and accesses a data base that holds the relationship between these address keys and sites, using the obtained keys. In this case, when the positional information (the coordinates on a topography) on the current site is given, the obtained pieces of site information (IDs, and addresses) are arranged, starting with the element closest to the coordinates. The above processes are repeated.

It is judged whether the retrieving at step SD8 is successful or not (step SD9). If the retrieving has failed, the process at step SD5 in FIG. 13D is performed. At step SD9, when the retrieving is successful, it is judged whether more than one piece of data has been retrieved or not (step SD10 in FIG. 13A). When one piece of data has been retrieved, the obtained site ID is set in the destination site and the process is terminated (step SD11 in FIG. 13C). At step SD10, when more than one piece of data has been retrieved, the address information is displayed in the site list on the display screen, starting at the closest site on topography (that is, in the order of pieces of data retrieved). Further, the effect that more than one candidate exists in the message area of the display screen is displayed and a message to prompt selection of a candidate is displayed (step SD12).

Next, it is judged whether the user has entered data by selecting an option from the site list (for example, selecting data using a mouse and a mouse button) (step SD13). When the user has entered data by selecting an element from the site list, the ID of the selected site is set in the destination site (step SD14). Further, the state of the site list is returned to the original display state, and the process is terminated. When judgment at step SD13 shows NO, the display of the site list is returned to the original state (step SD15), and the process is terminated.

At step SA11 in FIG. 9B, it is judged whether or not the user has entered data in the phone number field. If he has entered data in the phone number field, the phone number process is performed (step SA12). The phone number process is carried out using a retrieving agent with a phone number as a key in a similar manner to the postal code number process. When the retrieving is successful, the retrieved site ID is set in the destination site.

At step SA13, it is judged whether or not data has been entered in the destination field. If true, the destination input process is performed (step SA14).

Referring to FIG. 14, a concrete process of the where-to-go input process at step SA14 will be explained. FIG. 14 is a flowchart for the process of specifying a where-to-go site by language entry.

The language analyzing section makes a language analysis of the character string set in the where-to-go field (step SE1). At step SE1, using a dictionary containing the site's names, parts of speech, the types of site, and the site's IDs, ordinary morphemic analysis and syntactic analysis are made. In the analysis, for example, when "Tokyo Station" is entered, it is a proper noun, and its site ID is obtained from the information in the dictionary. When "Library" is entered, the information that it is a noun indicating a site is obtained. When "Setagaya-ku no toshokan" (a library run by Setagaya Ward) is entered, the information is obtained that the area site "Setagaya-ku" (Setagaya Ward) and the site "toshokan" is connected by the case particle "no."

Based on the analyzed information obtained at step SE1, retrieving of the actual where-to-go site ID is performed (step SE2). For entry of "Tokyo Station," the site ID is already obtained, since the site ID has been retrieved from the dictionary information. For "library," it is evident from the language analysis that it is a noun representing a site, and based on this information, the site ID is retrieved. For retrieving of the site ID, a specified number of sites are retrieved, sites adjacent to the current site first, then those adjacent but one, then adjacent but two, . . . . Among them, when there is a site corresponding to "library," the site is treated as a where-to-go site. After retrieving, when there is no corresponding site, a request that the "library" site in the area should be retrieved is issued to the area site containing the current site. When there is a corresponding site, the site is treated as a where-to-go site.

For entry of "Setagaya-ku no toshokan," a request that the "library" site in the area site is issued to the "Setagaya-ku" area site, according to the definitions in the input pattern table (not shown). When there is a corresponding site, the site is treated as a where-to-go site. By this process, the site ID can be obtained from the input character string to the where-to-go field. When retrieving of the site ID has failed, the failure is sensed at step SE3 and an error is displayed on the display screen and the process is terminated (step SE4). When more than one site ID has been obtained, the candidates are presented to the user in sequence, starting at the closest one to the current site, which allows the user to select one of them (step SE6). The site ID is set in the destination site and the where-to-go input process is terminated (step SE7).

At step SA15, it is judged whether or not the cancel button is selected. If pressed, the processes related to movement are terminated.

At step SA16, it is judged whether or not the execute button is selected. If not, the processes from step SA3 are repeated. When the execute button is selected, it is judged whether or not the destination site ID has been already set at step SA17. If true, the process of moving to the site ID is executed. When the destination site has not been set, that effect is displayed on the screen at step SA19 and the processes from step SA3 are repeated.

By the above operations, the user can move from the current site to another site.

In the above embodiment, site movement by address has been explained.

In this case, a virtual site that performs a process on behalf of a certain topographical area is effective. This type of site is called an area site. Area sites are created so as to correspond to regional divisions, such as the country, the prefecture, the city, the town, etc. They are connected and constructed so as to form a hierarchical structure of area sites. As an example, the area site of Setagaya Ward (referred to as the Setagaya-ward area site) is taken. At a higher level above the Setagaya-ward area site, the area site of Tokyo Station is connected under the relationship of "is-part-of," whereas at a lower level below the Setagaya-ward area site, the area sites of each town (such as Naka town or Seta town) in Setagaya Ward are connected.

FIG. 15 is a view of a display screen when a move is made to the Setagaya-ward area site. On the screen, a map for Setagaya Ward is displayed. The map is divided into the lower area sites of Setagaya Ward. Further, the Setagaya-ward area site has a site agent for questioning and answering persons in the site, the site agent appearing on the screen. Buttons for starting ward's administrative services and retrieving software for information on ward are displayed. The user can move from the Setagaya-ward area site to Naka-town area at a lower level, make an inquiry about a site in Setagaya Ward, and receive services such as result display.

Various functions related to the connection between persons (agents) will be explained. The connecting process is the process of setting communication channels between persons. Here, an example of person A corresponding to a certain user A and person B corresponding to a person B will be explained.

FIG. 16 is a view of an access screen to person B. In this example, the mouse cursor is moved onto the image of person B (Yamada) in site DB 210, the mouse button is pressed to display an access menu to person B. On the screen, the name of person B, the place where he is now, and the state of the other party appear. At the same time, The "ACCESS" select button appears which is used to determine whether accessing is achieved actually or not. Methods of accessing a person include a method of directly specifying a person in the site as shown in the figure, and a method of accessing a person from the person's home agent. In the latter case, the current site information on person B stored in the home agent is taken out, and message communication with the site establishes a connection with person B.

When sensing user A's access request to person B (for example, with the mouse button pressed), person A transfers to person B an access request message together with a message containing specified accessing person information. Person B, receiving this message, informs user B that person A has accessed.

Person B, depending on the information on the other party's person and its own condition, performs request arbitration processing to obtain information on person B to be presented to the other party and accessibility. In the request arbitration processing, depending on the conditions, after an inquiry is made of user B, judgment is made. The request arbitration processing, which will be explained later in detail in a second embodiment, judges degree of urgency of business relationships (superiors, colleges, subordinates, customers, good customers), personal relationships (friends, family, etc.), and the other party, and depending on the degree of urgency, creates a message to be sent to person A.

Person A presents to user A the person information and the accessibility information sent from person B. When the user corresponding to person A selects access to person B, person A issues to person B a request for selection of access method (moving picture channel, speech channel, text transmission channel, etc.). Person A, based on this information, performs channel connection to establish a specific communication channel between person A and person B. After the communication channel has been established, the communication display processing program is started. The relationships between the above processes are shown in FIG. 17.

FIG. 18 is a view of a screen on the user side corresponding to person A for establishing a communication between person A and person B, which is displayed by the communication display program.

The screen of this example contains an area 321 displaying the other party's name 321a, the current site 321b, and the state 321c. In an area 322 for displaying information on the other party, real-time moving pictures and still pictures previously prepared at the user corresponding to person B are displayed according to selection of the communication means. In the area 322, it is displayed whether what corresponds to person B is the user or not. In this example of the screen, the description AGENT indicates that the corresponding other party is not a human being.

An area 323 used for text communication with the other party (the called party) is divided in two: an area in which a call of the other party is displayed and an area in which a call of the calling party. When speech communication is possible, the area 323 can be used to convert vocal conversational give-and-take into text for display. Further, the screen of this embodiment is provided with an area 324 for transferring given data to person B, an area 325 for receiving given data from person B, an area 326 for synchronizing its movement with the other party's movement or synchronizing the other party's movement with its movement, an area 327 for sending information to the other party's home agent even if it is impossible to communicate with the other party, and an area for specifying the recording of communication state.

Referring to FIG. 19, various operations of the person and the workings and structure of the person will be explained. FIG. 19 is a block diagram of the person's response process.

A recording section 350 has the function of recording the input information from the other party's person and the output information to the other party's person in a storage unit (not shown). A correspondence switching section 360 supplies the input from the other party either to an automatic processing section 370 or to a user interface section 380. The correspondence switching section 360 supplies the output to the other party either from the automatic processing section 370 or from the user interface section 380. The automatic processing section 370 analyzes requests from the other party, and depending on the results, supplies the output to the other party or exchanges the data with the user via the user interface section 380. The user interface section 380 performs an input/output process with the user by means of input/output equipment.

The correspondence switching section 360 allows two output switching settings: K1 and K2. At initial setting, it is set to K1 leading to the automatic processing section 370. Taking the above example of person A and person B, the operation of the person will be explained. When there is an access request from the other party's person, the access request is sent to the automatic processing section 370 via the recording section 350 and the correspondence switching section 360. The automatic processing section 370 performs a connecting process. In the state where the connecting process has finished, the display screen appears at the user on the person A side, whereas the automatic processing section 370 presents to the user on person B side the fact that an access from person A has taken place.

In this state, when the user on person B side decides to make direct communication with the user on person A side, a switching request is issued to the correspondence switching section 360 via the user interface section 380. By this request, the input from person A is outputted to K2 and directly sent to the user interface section 380, with the result that display screen similar to that at the user on person A side appears on the screen on person B side. When there is no user switching request on person B side, the input from person A side is sent without change to the automatic processing section 370.

FIG. 20 is a block diagram of the automatic processing section 370 of FIG. 19.

A request analyzing section 371 analyzes the information sent from the correspondence switching section 360 and extracts a problem to be solved. A problem solving section 372 solves the problem extracted at the request analyzing section 371. A knowledge retrieving section 373 contains knowledge information the person has, and retrieves the contents according to the request from the problem solving section 372. A response generating section 375 judges whether or not the information obtained from the problem solving section 372 should be given to the requester. The response generating section 375 converts the information sent from a response judgment processing section 374 into a specified form, which is then sent to the correspondence switching section 360.

Referring to FIGS. 21A through 21C, the operation of the automatic processing section 370 with respect to various inputs from person A side will be explained. FIGS. 21A through 21C are flowcharts for the processing at the request analyzing section 371.

When the input from the person A side is sent from the correspondence switching section 360, the request analyzing section 371 judges whether the contents are a speech input or a natural-language sentence input (step SF1). At step SF1, if YES, the effect that the process is in progress is displayed on the display screen at the user corresponding to person A (step SF2 in FIG. 21B).

Next, a speech recognition process and a natural-language recognition process are carried out (step SF3). These processes are performed using a speech recognition process module and a natural-language recognition process module. The recognition results are classified into any of Greeting, Data Request, Data Notification, Direct Dialogue Request, End Request, and Recognition Failure. For example, when "Good afternoon" is entered, the result of analysis is "Greeting: type 1," according to the dictionary information in the natural-language processing module. Here, "type 1" indicates a greeting representing a cue for starting conversation. The sentence "Where is . . . ?" is analyzed as the problem of asking the location of . . . as shown in "Data Request: location (. . . )." An input such as "want to speak directly" is analyzed as Direct Dialogue Request, and an input such as "Goodbye" is analyzed as End Request.

Steps SF4 through SF14 are the processes of sending the analysis results at step SF3 to the problem solving section 372.

At step SF3, when the analyzing process has failed, the problem solving section 372 is informed that the contents of judgment that the analysis is unacceptable, such as "Analysis Failure: Inarticulate Speech." When the input from person A is neither a speech input nor a natural-language sentence input, step SF15, step SF16, and step SF18 judge whether or not it is Data Notification (transfer of text files or the like through f on the display screen), Interlocking Request, and End Request, respectively. Then, step SF9, step SF17, and step SF12 notify the request analyzing section 371 of the contents, respectively, and terminate the process. When the input falls into none of the above cases, the problem solving section 372 is notified of "The Other Requests" and the process is terminated.

FIGS. 22A and 22B are flowchart for the processing at the problem solving section 372.

When the request analyzing section 371 judges that the input from the other party's person is a greeting (step SG1), a particular response to it is selected (step SG2). This response is obtained by retrieving a table (not shown) previously stored in the knowledge retrieving section 373. The response judgment processing section 374 is notified of the retrieving result as an answer (step SG3). At step SG1, when the input is judged to be an input data request (step SG4), the knowledge retrieving process is performed (step SG5). In the knowledge retrieving process, the knowledge retrieving section 373, using the question as a key, retrieves the answer, For example, when to the question about location (business trip application) (Where is business trip applications?), "location (business trip application)=cabinet 1" is written in the data base in the knowledge retrieving section 373, simple matching gives the answer "cabinet 1."

At step SG5, when the same question is analyzed at its intention level by the request analyzing section 371 as "give (business trip application)" and the knowledge retrieving section 373 has such a description as "give(X)=retrieving program (X)", the retrieving program can be started using the business trip application as an argument, take out the description and make it an answer. When the retrieving process has given no answer, this means that the automatic process of a person request has not been performed as expected. Its state is presented to the user in a suitable form (step SG7). The response judgment processing section is informed of the result of the retrieving and the answer (step SG8). When the request analyzing section 371 has judged that the information is a data notification request, a direct dialogue request, an end request, an interlocking request, and other requests (step SG9), the information is notified as it is to the response judgment processing section 374 (step SG3).

When the request analyzing section 371 has failed in analyzing a request (step SG10), it is judged whether or not specified conditions including the number of successive analysis failures and its ratio are met (step SG11). At step SG11, when the specified conditions are fulfilled, it is judged that the dialogue process has failed, and this information is notified to the response judgment processing section 374 (step SG12).

At step SG13, depending on the cause of the analysis failure, it is judged whether or not an input request should be made to the other party. The causes of the analysis failures include low confidence at the speech recognition section, unknown words at natural-language analysis, more than one interpretation of a sentence, and other various factors. At step SG13, for these failure factors (or failure to identify a unique interpretation), it is judged using a previously specified table that what kind of measure should be taken against each factor. For example, when there are three possible candidates among the analysis results, they are presented to the other party's user to allow him to choose one of them.

At step SG13, when it is judged that an input should be made again, a reentry process is notified to the response judgment processing section 374 (step SG14). At step SG13, when it is judged that an input should not be made again, it is judged that the state where telephone conversation is difficult has developed, and its information is notified to the response judgment processing section 374 (step SG15). At step SG10, when it is judged that there is no analysis failure, the fact that the problem solving process has failed is notified to the response judgment processing section 374, and the process is terminated (step SG16).

FIGS. 23A through 23E are flowcharts for the processing at the response judgment processing section 374.

The response judgment processing section 374, depending on the result of solving a problem informed by the problem solving processing section 82, determines what response to return. As required, the input and output from and to the user is performed via the user interface section 380.

It is judged whether the notification from the problem solving section 372 is about a problem solving failure, a poor dialogue, a direct dialogue request, an interlocking request, or other requests (step SH1). In this case, it is desirable to ask the user for judgment.

A check is made to see if the use can deal with the situation (step SH2). In this case, when the user cannot access from the computer (because the computer is not used or is far away from the user) or when the user can access from the computer but the computer is processing a job of higher priority, control proceeds to step SH3, where the information on the fact that the user cannot deal with the situation is notified to the response generating section. When the user can access, the notified conditions from the problem solving section 372 are displayed on the user's display screen in a specified mode, and the process is terminated. When the user can deal with the situation, the notified conditions from the problem solving section 372 are presented via the user interface section 380 shown in FIG. 29 to ask the user for instructions (step SH4). In this case, when the user denies intervention, the information on the fact that the user has denied intervention is notified to the response generating section 375 and the process is terminated (step SH6).

When the user has instructed switching of the correspondence process from the automatic processing section 370 to the user, a switching request is sent to the switching section 360 (step SH8). By this request, the switching section 360 switches the input/output from and to the other party's person from the automatic processing section 370 to the user interface section 380. As a result of this switching, the screen display on the user side is connected to the other party's person, allowing communication between two persons. While the automatic processing section 370 is dealing with this situation, this exchanging state is displayed in a reduced mode on the user's screen on the person B side, for example. After this switching process, the display is changed to a specified size. When the user has not specified the correspondence switching process (step SH7), the user enters an answer to the problem solving process (step SH9), and the information is notified to the response generating section 375 (step SH10). At step SH11, it is judged whether or not the notice from the problem solving section 372 is that the problem has been solved successfully.

At step SH11, when the problem has been solved successfully, a security check is made on the obtained answer data (step SH12). The security check starts with the security level of the other party's person, the access qualification, the security level of the data, etc. When they do not pass the security check, the user is asked to present data (step SH13). If the user refuses to present data or an inquiry cannot be made of the user, it is reported that it is difficult to present data, and the process is terminated (step SH15). When they have passed the security check, and at step SH14, when the user has approved, the answer data is notified to the response generating section 375, the process is terminated (step SH16). At step SH17, it is judged whether the notice from the problem solving section 372 is of data notification or not.

At step SH17, it is of data notification, the information that data has been received is presented (if possible) to the user (step SH18), the received data is held (step SH19), it is reported that the data has been received (step SH20), and the process is terminated. At step SH21, it is judged whether or not the notice from the problem solving section 372 is of an end request. When it is of an end request, end is notified to the response generating section 375 (step SH22), and the end process is performed (step SH23). In the end process, the end of the response process at the response generating section 375 is confirmed. Further, it is confirmed that there is no input from the other party's person in a specified period of time, the reserved communication line is canceled and the user screen is erased.

At step SH24, it is judged whether or not the notice from the problem solving section 372 is of reentry. If it is of reentry, selection of a message to be presented to the other party's person is made based on the cause of a failure at the problem solving section 372, and this is notified to the response judgment processing section 374. Therefore, this is notified to the response generating section 375 (step SH25), and the process is terminated. At step SH26, it is judged whether or not the notice from the problem solving section 372 is of communication trouble. If true, a response trouble message is selected (step SH29), and control goes to step SH2. At step SH27, a communication trouble message is selected, and at step SH28, this is notified to the response generating section 375, and the process is terminated.

The response generating section 375 creates a return message to the other party's person on the basis of the information reported from the response judgment processing section 374. The return message is sent to the other party's person via the correspondence switching section 360. Table 1 and 2 show examples of an output sentence created by the response generating section according to the notified information.

                  TABLE 1
    ______________________________________
    Description    Illustrative sentence
    ______________________________________
    Greeting       A greeting message obtained at the
                   problem solving section 372
                   Good afternoon etc.
    Problem solving
                   Message such as "I can't say
    failure        anything definite about it unless
                   asking the user."
    Dialogue trouble
                   "We can't put you through to him
                   now. What would you like us to
                   do?"
    Direct dialogue
                   "He is now in conversation and
    request        cannot have a talk with you."
    rejection
    Interlocking
    request rejection
    Data presentation
                   "This case requires the
    trouble        words. . ."
    ______________________________________


              TABLE 2
    ______________________________________
    Description    Illustrative sentence
    ______________________________________
    Data presentation
                   The data required is actually
    possible       presented.
    Data notification
                   "The data sent is received."
    received
    End notification
                   "Goodbye" etc.
    Reentry request
                   "Speak more slowly again."
                   "I can't understand. . .  Would you
                   repeat it?" etc.
    Response trouble
                   "I am sorry I can't understand. I
                   am also in communication trouble."
    ______________________________________