Distributed document-based calendaring system

Abstract

A system and method for receiving, analyzing, and managing a database of calendar information obtained from a variety of source documents. Information derived and extracted from calendar documents is used establish a database of events. Extraction of relevant information from the calendar documents is facilitated by guidance in the form of user input, document templates, or other specification of the structure and content of the calendar documents.

Claims

What is claimed is:

1. A method for manipulating and managing a calendar based on information contained in at least one source document, comprising:

receiving data representative of at least a portion of not less than one source document;

receiving guidance representative of the genre of the not less than one received source document;

selecting an input specification based on said genre of the not less than one received source document;

using said input specification to guide identification of data of interest in the not less than one received source document;

extracting said identified data of interest from the not less than one received source document;

merging said extracted identified data of interest into a distributed genre, said distributed genre including specification of source document input genres, specification of output genres, identified pieces of data of interest still resident in source documents, mapping of data of interest between the source documents and at least one output document, an explanation of social context, and a computational engine;

selecting an output specification, wherein said output specification identifies a selected output genre; and

creating at least one output document in said selected output genre, wherein said output document is created from said extracted identified data of interest merged within said distributed genre.

2. The method of claim 1, wherein the source document comprises a bitmap image, and wherein the identifying step comprises performing character recognition on the bitmap.

3. The method of claim 2, wherein the using step comprises the steps of:

identifying at least one date; and

identifying at least one event title.

4. The method of claim 3, wherein the step of identifying at least one date comprises finding a portion of the data consistent with an exemplary date representation.

5. The method of claim 3, wherein the step of identifying at least one date comprises receiving user input representative of a location within the data.

6. The method of claim 1, further comprising receiving user input representative of at least one location within the data.

7. The method of claim 6, wherein the data comprises a bitmap of the source document, and wherein the using step comprises the steps of:

performing character recognition on a portion of the data near the location;

locating at least one data item within the portion.

8. The method of claim 7, wherein the data item comprises a date.

9. The method of claim 7, wherein the data item comprises an event title.

10. The method of claim 6, wherein the user input comprises data from a scanning pen.

11. The method of claim 6, wherein the user input comprises data representative of a written annotation.

12. The method of claim 1, wherein the data comprises a digitization of an audio message, and wherein the analyzing step comprises performing voice recognition on the digitization.

13. The method of claim 1, wherein the guidance comprises user input.

14. The method of claim 1, wherein the guidance comprises a template for the source document.

15. The method of claim 1, wherein the guidance comprises a genre specification for the source document.

16. A calendaring system for manipulating and managing a database on information contained in at least one source document, comprising:

an input device adapted to receive the document;

means for receiving guidance representative of the genre of the received source document;

means for selecting an input specification based on said genre of the received source document;

means for using said input specification to guide identification of data of interest in the received source document;

processor means, comprising:

means for extracting said identified data of interest from the received source document;

means for merging said extracted identified data of interest into a distributed genre, said distributed genre including specification of source document input genres, specification of output genres, identified pieces of data of interest still resident in source documents, mapping of data of interest between the source documents and at least one output document, an explanation of social context, and a computational engine; and

means for selecting an output specification, wherein said output specification identifies a selected output genre; and

an output device adapted for creating at least one output document in said selected output genre, wherein said output document is created from said extracted identified data of interest merged within said distributed genre.

17. The calendaring system of claim 16, wherein the calendar information comprises at least one event.

18. The calendaring system of claim 17, wherein the event comprises a date and a title.

Description

FIELD OF THE INVENTION

The invention relates to an information management system and method, and more particularly to a system and method for receiving, analyzing, and managing calendar information obtained from a variety of source documents.

BACKGROUND OF THE INVENTION

Today, the typical setting for a personal computer system is still an office. Since personal computers started becoming prevalent over twenty years ago, many of them have been used for such applications as word processing (and other document preparation), financial calculations, and other office-related uses. They have not permeated the home environment, other than for games and for displaced office-type work, because they are not simple to operate.

During that time, the primary user-interface paradigm for interacting with computers has been a keyboard-and-screen system. Although this arrangement has been improved and refined over the years, it is still essentially the same arrangement that has been used with computers for many years, and was used on remote terminals even before the advent of personal computers.

The keyboard-and-screen system presents several advantages. The keyboard typically used with computer systems is an inexpensive item to produce. It includes only around 100 key switches, signals from which are encoded and sent to the CPU. Also, it is only slightly modified from the version used on mechanical typewriter keyboards for over a century. Hence, it is familiar to most people. Moreover, typewriter-style keyboards (or variations thereof) are usable to unambiguously input information in most Western languages.

However, for most people, a keyboard is not an efficient form of input. To use a keyboard effectively, training is required. Even with the requisite training, time and effort is necessary to enter information via keyboard, particularly when the information sought to be entered is already evident in a document or some other communication from another. Moreover, they are sensitive to spelling errors, repetitive stress injury (such as carpal tunnel syndrome and tendinitis), and inconvenience. Both hands are needed to use a traditional keyboard with any speed.

The display screens (or "monitors") typically used on personal (and other) computer systems have a number of advantages. They are relatively inexpensive. Years of producing television sets and computer monitors have resulted in manufacturing and design efficiencies and improvements in quality.

However, even with their improvements, CRT-based display screens are still typically bulky, heavy, and energy inefficient. They also produce a relatively large amount of heat. For these reasons, CRT displays have not been integrated into many other environments, and computers (and their displays) are usually treated as stand-alone items. Other display technologies have been tried, including plasma displays and liquid crystal displays (LCDs), to name two, but have been less successful because of their relatively high cost and low image quality in comparison to CRT displays. However, LCD prices have been dropping in recent years, and such displays are beginning to be found in a variety of applications.

While the keyboard-and-screen scheme for interacting with computers has proven to be satisfactory in many ways for a long time, there are some problems that are not easily resolved with such a system. For example, there can be a lack of correlation between what is displayed on the screen and what is entered on the keyboard. Any formatting information available on the screen must be entered via sequences of keystrokes on the keyboard, and those sequences in many cases are not intuitive. Furthermore, many symbols and items viewable on the screen can not easily be entered via keyboard.

Recently, however, progress has been made in the usability of alternative user interface schemes. For example, touch-screen-based systems, in which a flat-panel display (such as an LCD) is overlaid with a translucent pressure-sensitive (or other type of touch-sensitive) surface, have been gaining in popularity. Such a system allows the user to directly manipulate the information that is shown on the display. For example, various gestures can be made on the surface to copy, move, annotate, or otherwise alter information on the display. Where such a system falls short, however, is in data input. Where there is no keyboard associated with a touch screen, then data must be input via interaction with the touch-sensitive surface. In some cases, this involves handwriting recognition, which is an imperfect and computationally intensive procedure, or some other means, such as "pressing" (with a stylus or fingertip) a visually displayed keyboard, or by special gestural symbols designed for data entry.

Voice recognition input has also made some progress in recent years. In the past, voice recognition systems have been used primarily in experimental environments. Typically, error rates were extremely high, and to accomplish real-time recognition, the computational resources required were prohibitively high. Recently, however, several commercial software products have made it possible to offer real-time voice recognition on personal computers of the type frequently used in the home. However, such voice recognition systems are speaker-dependent, and as such require a significant amount of training to attain a satisfactory level of performance and a low enough error rate. Moreover, when errors are made (such as in the recognition of homonyms and proper names), it is frequently more convenient to type the corrected word with a traditional keyboard than it is to correct the error by speaking the necessary voice commands and spelling the word for the system. Accordingly, voice recognition shows some promise for the future, but at the present time, is not a practical method of operating and providing input to a personal computer.

Despite promises of cross-plafform integration (e.g., computer and television, computer telephony), there is usually little relationship between the data on a personal computer and most of the documents and other tools used for communication and information exchange that are found around a typical individual, office, or family. For example, in a typical home or office, one might find a telephone, an answering machine (or voicemail system), audio equipment (such as a stereo), a fax machine, a television, a computer and printer, a whiteboard or a chalkboard, and various written notes, lists, calendars, mailings, books, and other documents. Unfortunately, the information in one or more of those repositories is usually tied to that repository. For example, addresses in a written address book are not easily used on a computer e-mail system, unless the user goes to the trouble of manually transferring the relevant information from the address book to the computer.

Furthermore, there is a well-known lack of compatibility between systems of different types, even those systems that are designed to work together. For example, in the conversion between one data format and another, there may be a loss of formatting or other information. Furthermore, errors may creep into the conversion, as when optical character recognition (OCR) is used to convert a printed document to a machine-readable one.

Because of these obstacles, the numerous disparate data types and formats persist in the home and office environments. For example, written notes on a family's refrigerator door are frequently a useful and convenient means of communication. The kitchen is often a place of gathering, or at least a place where each family member will visit several times every day. Accordingly, when one family member wishes to communicate with another that he might not see in person, then he might write a short note and post it to the refrigerator door with, for example, a magnet. Other documents, such as calendars, computer printouts, facsimiles, and collaborative lists can also be posted to the refrigerator door.

Several companies have introduced limited-function kitchen computers, or software for general-purpose personal computers to enable kitchen functionality. Such kitchen computers usually provide the ability to store recipes, create shopping lists, and take rudimentary notes. However, in most cases, these kitchen computers use the standard keyboard-and-screen user interface, and are highly limited in function. Kitchen computers generally do not have very well-developed document handling or telephony functions.

A class of portable computers known as personal document readers (or PDRS) has also arisen in recent years. The goal of these devices is to serve as a replacement for a printed book. Accordingly, a typical PDR is relatively light in weight, but has a large high-contrast screen to enable easy reading. Recent PDRs also have other capabilities, such as the ability to annotate a document (either via a built-in keyboard or a touch-sensitive screen adapted for writing with a stylus). Known PDRs are generally limited in function, but as will be discussed below, can frequently be used as an input/output terminal in an embodiment of the invention to be disclosed herein.

Another class of systems uses what is known as a "paper user interface," in which commands are conveyed to the system by making marks on paper, which is then scanned. For example, one category of such devices is able to read free-form ink (or digital) annotations to determine which of several possible editing operations a user wishes to perform on a document. See, e.g., U.S. Pat. No. 5,659,639 to Mahoney and Rao, entitled "Analyzing an Image Showing Editing Marks to Obtain Category of Editing Operation." Other versions of "paper UI" systems are capable of interpreting drawn symbols as commands, deriving commands from marked-up forms (e.g., checkboxes) attached to a scanned document, and reading pre-printed one- or two-dimensional data codes (such as Xerox DataGlyphs). For a summary of the state of the art in this area, see, for example, U.S. Pat. No. 5,692,073 to Cass, entitled "Formless Forms and Paper Web Using a Reference-Based Mark Extraction Technique."

Such techniques can also be extrapolated to other media, such as office whiteboards. See, e.g., U.S. Pat. No. 5,528,290 to Saund, entitled "Device for Transcribing Images on a Board Using a Camera Based Board Scanner," and U.S. Pat. No. 5,581,637 to Cass and Saund, entitled "System for Registering Component Image Tiles in a Camera-Based Scanner Device Transcribing Scene Images." An all-electronic system for accomplishing essentially the same result is Tivoli, an electronic collaboration tool that uses Xerox LiveBoard hardware. See Pedersen, E. R., McCall, K., Moran, T. P., and Halasz, F. G., "Tivoli: An electronic whiteboard for informal workgroup meetings," Proceedings of the InterCHI'93 Conference on Human Factors in Computer Systems. New York: ACM (1993).

While all of the foregoing systems are beneficial and useful in certain limited situations, they are all directed to solve limited problems. Accordingly, while they may be useful in an office setting, they might not easily transfer to other settings. Accordingly, there is a need for a document and information management system that is easier to use than traditional systems, yet powerful enough to be adaptable to numerous situations. Such a system should simplify the user's work, even if it does require some input and assistance. It should be able to handle documents and input in a variety of formats, structures, and media, including printed, written, and spoken communications.

SUMMARY OF THE INVENTION

This invention builds upon the limited successes of prior systems in an attempt to create a comprehensive document handling system and method, useful in home and office environments, that is intuitive, easy to use, powerful, and relatively ubiquitous by way of its incorporation into other traditional channels of communication.

It requires no structural changes to its source documents, yet it is able, with minimal assistance, to extract information for use in an information database system. It is capable of accepting input from a large number of sources, including documents in the physical and digital domains, and in many different media types, including printed documents, handwriting, audio messages, and electronic messages. To do this, the system and method of the invention rely upon the analysis of information from multiple sources, including, when necessary, limited user input. The end result is a product that is usable in either digital or physical form, breaking down the barriers between the digital and physical document worlds, and allowing essentially all types of information to be exchanged with a minimum of. difficulty.

The invention relies upon the recognition and analysis of document genre structure rather than content. The document genre guides the extraction of useful information, while reducing the need to recognize and parse each document in its entirety. This reduces errors and computational expense.

Accordingly, a method according to the invention for manipulating and managing a calendar includes the steps of receiving data representative of at least a portion of a source document, receiving guidance representative of the source document's genre or structure, analyzing the data, using the guidance to identify an event within the data, extracting a representation of the event from the data, and merging the representation into a database. This method is capable of operating with or without user interaction.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating various exemplary physical components of a networked document processing and database system according to the invention;

FIG. 2 is a block diagram illustrating various exemplary functional components used in a network according to FIG. 1;

FIG. 3 is a block diagram providing a high-level functional overview of an information management system according to the invention;

FIG. 4 is a flow chart illustrating the steps performed in an exemplary calendaring system according to the invention;

FIG. 5 is a flow chart representing the steps performed in an automated input analysis step as performed in the method set forth in FIG. 4;

FIG. 6 is a flow chart representing the steps performed in a semi-automated input analysis step as performed in the method set forth in FIG. 4;

FIG. 7 is an exemplary school schedule capable of being processed by the calendaring system of FIG. 4;

FIG. 8 is an exemplary program schedule capable of being processed by the calendaring system of FIG. 4;

FIG. 9 is an exemplary school snack schedule including superfluous information capable of being processed by the calendaring system of FIG. 4;

FIG. 10 is an exemplary soccer game schedule capable of being processed by the calendaring system of FIG. 4;

FIG. 11 is an exemplary wedding invitation capable of being processed by the calendaring system of FIG. 4;

FIG. 12 is an exemplary electronic output calendar capable of being generated by the calendaring system of FIG. 4;

FIG. 13 is a flow chart illustrating the steps performed in an exemplary telephone message processing system according to the invention;

FIG. 14 is a diagram illustrating the typical structure of a telephone message.

FIG. 15 is a flow chart representing the steps performed in an automated message analysis step as performed in the method set forth in FIG. 13;

FIG. 16 is a flow chart representing the steps performed in a semi-automated message analysis step as performed in the method set forth in FIG. 13;

FIG. 17 is a flow chart representing the steps performed in a primarily manual message analysis step as performed in the method set forth in FIG. 13;

FIG. 18 is a diagram illustrating the typical structure of a spoken telephone number;

FIG. 19 is a flow chart illustrating the input steps performed in an exemplary distributed genre document processing system according to the invention;

FIG. 20 is a flow chart illustrating the output steps performed in an exemplary distributed genre document processing system according to the invention;

FIG. 21 is a diagram illustrating the typical structure of a spoken or written date;

FIG. 22 is a diagram illustrating the typical structure of a spoken or written time of day;

FIG. 23 is a diagram illustrating the typical structure of a location or address;

FIG. 24 is a functional block diagram illustrating the components of an exemplary moded scanning pen according to the invention;

FIG. 25 is a visual representation of an exemplary moded scanning pen according to the invention having a first form factor;

FIG. 26 is a visual representation of an exemplary moded scanning pen according to the invention having a second form factor;

FIG. 27 is a visual representation of an exemplary mode book for use with a moded scanning pen according to the invention;

FIG. 28 is a functional block diagram illustrating the components of an exemplary parasitic user terminal according to the invention;

FIG. 29 is a visual representation of the parasitic user terminal of FIG. 28 mounted to a host refrigerator; and

FIG. 30 is a visual representation of the parasitic user terminal of FIG. 28 mounted to a wall.

DETAILED DESCRIPTION OF THE INVENTION

The invention is described below, with reference to detailed illustrative embodiments. It will be apparent that the invention can be embodied in a wide variety of forms, some of which may be quite different from those of the disclosed embodiments. Consequently, the specific structural and functional details disclosed herein are merely representative and do not limit the scope of the invention.

Referring initially to FIG. 1, a distributed network for information management and sharing according to the invention is shown in schematic form. As will be described in further detail below, the present invention is adapted to facilitate the extraction and use of significant information in documents of many kinds, including (but not limited to) papers, handwritten notes, business cards, e-mail messages, audio messages, and the like, without any appreciable advance knowledge of the content or format of the documents, but with some knowledge of the "genre" or context of the documents. As will be apparent from the description below, the system is adapted for distributed access and either centralized or distributed processing.

As used herein, the term "document" refers to any persistent communication or collection of information, whether fixed in a tangible medium (such as a hardcopy) or stored electronically, and whether in machine-readable or human-readable form. A document "genre" is a culturally defined document category that guides the document's interpretation. Genres are signaled by the greater document environment (such as the physical media, pictures, titles, etc. that serve to distinguish at a glance, for example, the National Enquirer from the New York Times) rather than the document text. The same information presented in two different genres may lead to two different interpretations. For example, a document starting with the line "At dawn the street was peaceful . . . " would be interpreted differently by a reader of Time Magazine than by a reader of a novel. Below (and in conjunction with FIGS. 7-12), a variety of calendars will be discussed; each one represents a different instance of the calendar genre. As will become clear from the discussion below, each document type has an easily recognized and culturally defined genre structure which guides our understanding and interpretation of the information it contains. That structure is used as the basis of certain aspects of this invention.

Two user terminals 110 and 112 each with a flat-panel display, a remote CPU 114, a traditional personal computer 116, a telephone device 118 with integrated telephone answering device (TAD) functionality, a document scanner 120, a fax machine 122, a printer 124, and a handwriting input tablet 126 are all coupled to a communications network 128. The network 128 may be any type of known network, such as an Ethernet local-area network (LAN), a wireless LAN (either RF or infrared), a wide-area network (WAN), or even the Internet. Moreover, the network 128 may comprise only the illustrated devices, connected in a peer-to-peer topology, or may include numerous other disparate devices, whether or not designed to operate with the present invention. As will be appreciated by individuals of skill in the art, numerous other network topologies and protocols may be used in an implementation of the current invention without departing from its scope. However, the functional interoperation of the illustrated devices will be considered in detail below.

FIG. 2 is a functional block diagram illustrating the functions performed by the physical components set forth in FIG. 1. The network 128 (FIG. 1) is provided in the form of a bi-directional communications link 210, to which a central processing unit (CPU) 212 and memory are attached. The CPU 212 is adapted to perform most of the invention's processing, but it should be noted that in an alternative embodiment of the invention, processing may be distributed over the network 128. Preferably, the CPU 212 is attached to a memory 214, which may be used to store, among other things, genre specifications used in document analysis (explained below), character models used for character recognition, voice models used in speech recognition, and other data used by the system.

A display 216 is also provided; it may be local to or remote from the CPU 212, and may be able to decouple from the network 128 for portability (as in a Personal Digital Assistant or PDA). An exemplary embodiment of the display 216 will be described in further detail below. Long-term storage 218 for the information database of the invention, which stores document information for later use, is also provided and connected to the communications link 210. Various input devices are attached to the link 210, including a keyboard 220, a pointing device 222 (such as a mouse, trackball, or light pen), a handwriting tablet 224, and a scanner 226. These devices are adapted to receive information and transmit it to the CPU 212 for processing.

It should also be recognized, however, that certain document types need not enter the system through any of the foregoing input devices. For example, an e-mail message received by the PC 116 need not be converted into the digital domain, as it is already in electronic form. The same is true for facsimile message; however, the latter may still need to be converted from a bitmap into a machine-readable code.

An audio interface, including a microphone 228 and a loudspeaker 230, facilitate the entry and use of audio documents (such as recorded memos). As suggested by FIG. 1, the microphone 228 and loudspeaker 230 may be integrated into a telephone device 118 or any other convenient apparatus attached to the network 128.

A printer 232 is provided for hardcopy output. It should be recognized that the foregoing input and output devices are exemplary only, and numerous other input devices 234 and output devices 236 may be used in accordance with the invention. Moreover, additional processors, such as a recognition processor 238 or any other processor 240 may be used to off-load some of the computational burden from the CPU 212. It is contemplated that the recognition processor 238 would be used to process audio from the microphone 228, handwriting from the tablet 224, or printed documents from the scanner 226, for example. In each case, the raw input would be translated to a machine-readable counterpart (e.g., speech would be converted to recognized text via a speech recognition algorithm, handwriting would be converted to recognized text or command gestures via a handwriting recognizer; or printed documents would be converted from a bitmap to text via optical character recognition).

The basic function of the overall system is illustrated by the block diagram of FIG. 3. A database 310 (preferably hosted by the storage 218 and managed by the CPU 212 of FIG. 2) serves as a repository of document information, specifically that information which has been deemed to be significant in documents processed by the system, and is coupled to an input/output subsystem 312, functionally illustrated in FIG. 2 above. The input/output subsystem 312 may include some or all of the display 216, the keyboard 220, the pointing device 222, the handwriting tablet 224, the scanner 226, the microphone and speaker 228 and 230, the printer 232, other input and output devices 234 and 236, as well as the logic used to control those devices, including the recognition processor 238, any other processor 240, and certain functions of the CPU 212. The input/output subsystem 312 is capable of handling documents 314, audio messages 316, and annotations 318, in accordance with detailed methods set forth below.

Input to and output from the database 310 are processed by the system with the assistance of guidance provided by a set of genre specifications 320 (preferably stored within the memory 214 of FIG. 2). The genre specifications 320 provide information on the expected nature, structure, and content of documents handled by the system. In particular, for input documents, the genre specifications 320 preferably indicate where certain data items are likely to be found within a document, the format of those data items, and other considerations of utility in the information extraction process. With regard to output documents, the genre specifications preferably indicate how to assemble data from the database 310 into human-readable documents in a manner consistent with what the user expects. The mechanics of this interaction will be considered below with reference to several detailed examples.

Calendaring System

A specialized application of the general system of FIGS. 1-3 is set forth and illustrated in FIG. 4. FIG. 4 represents the processes performed in a system adapted to extract appointment and other date/time information from various documents and inputs to the system, and to synthesize a calendar from the information then stored in the database 310 (FIG. 3).

Date and annotation information 410, stored as part of a calendar document, is an input to this method, which begins by receiving the date and annotation information (step 412). As suggested above, this input can occur in any of a number of ways: by scanning one or more paper documents with a sheet-fed scanner or a scanning pen, by selecting certain e-mail messages or Web pages for processing, or by providing an audio recording of a voice message, for example. A collection of one or more source documents is provided to the system. Consequently, the source documents can be, for example, paper or other physical documents, or can also be "raw bits," such as unrecognized image or speech data, or formatted data, such as ASCII or HTML. Each source document is assumed to represent a genre structure recognizable by the system. Each source document is further assumed to contain one or more key pieces of information, each such key being associated with a message that the user (or another person) would find significant. For example, if the genre is associated with a particular sort of calendaring event (such as a schedule) then the keys can be dates or times, and the messages can be announcements of scheduled events corresponding to these dates or times.

Typically, and importantly for the average user, the source documents are provided to the system in physical rather than digital form. In this case, the system captures the documents and converts them to digital form. For example, a paper document can be optically scanned by a whole page scanner or a scanning pen to produce unrecognized image data, which can then be further converted as by segmentation, optical character recognition (OCR), sketch analysis, and the like. An audio recording can be digitized and, if the recording contains speech, the speech can be recognized by a speech recognizer. Depending on the particular system, the documents may include data in any number of forms, such as textual, graphic, pictorial, audio, or video data. Streaming and static data are both acceptable. The greater the system's ability to accept physical rather than digital documents as input, the better it can integrate seamlessly with everyday practice.

At the time of document input, the genre of the document input is determinate, and is observable by the user. The document's genre is used to select a particular input specification 414, which is used to assist further processing of the input document. As will be appreciated by individuals of skill in the art, various means of specifying an input genre are possible; one exemplary method is described below with reference to FIG. 27. Moreover, the system may be programmed to expect documents having a particular genre. It is also possible for a system according to the invention to attempt to heuristically determine the genre of an input document without further user input. This is particularly possible when documents of different types and genres are being received from different sources (e.g., e-mail messages might usually have a certain genre, while facsimile messages might have a different usual genre).

For example, a particular input document may represent the genre of calendars. The characteristics of this genre are indicated by the selected input specification 414, chosen from a library of possible genre specifications. While, in general terms, a system according to the invention is able to process many different document genres, it is important to note that a single instance of the system may be equipped to only process a single genre type, such as calendars. Even so, there may be many different kinds of calendars, such as schedules, appointment books, and the like (some of which will be discussed in further detail below), all of which may be defined by a single input specification 414 or, if necessary, by multiple input specifications. For purposes of this discussion, all documents within the calendar genre are considered to have similar structures, including the key information set forth above.

The input specification 414 is employed to analyze the input (step 416) and identify information of interest. This operation may occur automatically (see FIG. 5) or semi-automatically (see FIG. 6) with some user interaction. The identified information (which for the calendar genre typically includes at least a date, a time, and an event title) is then extracted and associated (step 418) into a record corresponding to a single event. The record is then merged (step 420) into the database 310 (FIG. 3).

Alternatively, and preferably, the entire input document (or as much as is available) is merged into the database 310 and is indexed and referenced by its extracted records. This facilitates the ability to "look behind " the extracted event for additional information, if it proves to be necessary or desirable to do so. For example, an exemplary output calendar (see FIG. 12) may contain only a summary of information obtained from one or more input documents. When this is the case, the user can be given the opportunity to "select" an event, thereby revealing further information, possibly including a digitized bitmap of the entire input document from which the event was extracted. This capability provides an useful safeguard, reducing the possibility (and impact) of an error by the system.

When the user wishes to create output, an output specification 422 is selected. The output specification identifies and describes the characteristics of the user's desired output genre. For example, the user may wish to create a monthly calendar containing all family events for the month of December, 1998. The output specification 422 fully describes the format of that document, but does not contain any of the information from the database 310. Accordingly, the document is created (step 424) from the information in the database 310 and the output specification 422 and is outputted (step 426) to the user.

Given the automatic or semi-automatic processing of document information, it is entirely possible that the system failed to correctly identify the proper dates and times, for example. Accordingly, the user is given an opportunity to review the output document and to indicate to the system whether it is correct (step 428). If not, the analysis is adjusted (step 430), and information is extracted once again. As will be appreciated by individuals of ordinary skill in the art, this can be accomplished by several means, including but not limited to adjusting the parameters used to perform the analysis, by reverting to alternate choices in a weighted list of possibilities, or by accepting user guidance. To facilitate changing the analysis, it is contemplated that the database 310 continues to contain full document instances, in addition to the analyzed information derived in steps 416-418. Although the adjustment step 430 is presented herein as occurring after output has been generated, it should be noted that adjustment by any of the means set forth above can occur in any step of the process, for example immediately after character recognition is performed, or before the information is stored in the database.

FIG. 5 illustrates the process followed by an automatic version of the input analysis aspect of FIG. 4. Beginning with an input specification 510 and input data 512, character recognition (step 514) is performed on the input data 512, if necessary to translate the document into a machine-readable format.

From the recognized input, the portions of interest are identified (step 516). As will be appreciated by individuals of skill in the art, there are many ways to accomplish this; one method includes simply scanning for numerals, while another method scans for all potentially relevant types of information, such as numerals, names of months, names of days of the week, the phrase "o'clock," etc. From the portions of interest, the system then extracts the necessary dates (step 518), times (step 520), and event titles and other annotations (step 522), and if necessary, the applicable person (step 524). Dates and times, in particular, have reasonably well-defined structures that assist the system in identification. These characteristics will be discussed in further detail below (see FIGS. 18, 21, 22, and 23). In a household or business, names might also be readily identifiable (for example, by checking against a list of valid names).

Referring now to FIG. 6, which illustrates a semi-automatic version of the input analysis step of FIG. 4, a date 610, a time 612, an event title 614, and a person 616 are all identified by the user. Then, to facilitate extraction and indexing into the database 310 (FIG. 3), character recognition 618 is performed on the date 610, time 612, title 614, and person 616.

Clearly, in any particular document, at least one of the foregoing data items may not be present; for example, in a calendar might not include a time for an all-day event. In this case, the system may allow for the manual input of the omitted data item, or may alternatively continue to operate without it. As will be discussed in further detail below, the user identification of the data items 610-616 may operate by writing guiding annotations on a hardcopy document (which is then scanned), by using a scanning pen to input data fields in succession, or by any other similar method. In a more automatic alternative embodiment, user identification only requires "pointing at" a particular item; the system then uses character or pattern recognition to determine the extent of the written data field.

The manual input called for above may be accomplished via a keyboard, handwriting recognition, or simple selection of options via a pointing device, as circumstances require.

A detailed example of the calendaring system in operation will now be presented. It shows how a user can use one particular input mechanism, in this case a smart scanning pen (one embodiment of which will be discussed in further detail below), to analyze structure and merge calendar data from a variety of document genres. Note that as discussed above, a scanning pen is not the only possible means of input. Rather, the scanning pen is used for ease in describing the invention, which deals more with the source documents once they have been acquired.

In the particular embodiment described in this section, the user interacts with a smart scanning pen augmented with a single button or any other user control (one embodiment of which is illustrated in FIG. 24)--this button, when clicked, indicates to the system that a new appointment is being entered. The user's primary interaction with the scanning pen, once the button has been clicked, is to perform a series of "swipes" with the scanning pen across the surface of the document. Each such swipe yields a bitmapped portion of a text page. This bitmapped portion is then. analyzed and converted to digital text via optical character recognition (either on the pen itself, as part of the transfer to the system, or possibly later in the process by another component of the system), and analyzed with genre-specific heuristics.

The set of scanned events is transferred from the pen's local storage to the calendar system. A variety of standard mechanisms are available to accomplish this. In one embodiment of the invention, the scanning pen bears an infrared (IR) transmitter near the end farthest from the scanning tip. When the pen is "flicked" (or quickly pointed) in the direction of a host coupled to the network 128, the information is transmitted to the host (which must have an IR receiver ready to receive said information). Less exotic solutions are also possible, including but not limited to a docking station or even a serial cable.

An example of the scanning pen in use in the calendaring system operating as set forth in FIG. 4 is provided below. A click of the pen's button is indicated with the symbol CLICK, and a swipe of text is indicated by enclosing the text in double-quotes.

An exemplary school schedule is shown in FIG. 7. Note that the structure is of the schedule is self-apparent: it is a schedule of times when something (in this case school) is "on" and times when it is "off". Several additional refinements of this structure are present, including the application to particular subgroups (e.g., grades 1-3) or particular ways of being "off" (e.g., holidays). Suppose that a user wishes to enter the daily dismissal times into the system. Then an exemplary scanning sequence to accomplish this could be:

CLICK

"Regular Daily" (710)

"Dismissals" (712)

"2:45 PM" (714)

CLICK

"Every Wednesday Starting Sep. 3, 1997" (716)

"Dismissal for Grades 1-5" (718)

"1:30 PM" (720)

From the first swipe 710, "Regular Daily," the system can determine that the time of the appointment recurs daily--that time, furnished in the third swipe 714, is 2:45 PM. The system treats the second swipe 712 as an annotation for the event, as its data is not recognizable as either a date or a time.

From the fourth swipe 716, the system can determine the frequency and day of the second appointment. The fifth swipe 718 annotates that appointment, and the sixth and final swipe 720 provides the recurring time.

FIG. 8 illustrates an exemplary program calendar. This type of calendar is similar to the school example above (FIG. 7) in that it sets forth another schedule, this time oriented around the 12-month calendar rather than canonically. Suppose that the user wishes to enter into the system both events for September 816 and 822 and the Chanukah brunch 828 in December. A scanning sequence to accomplish this could be:

CLICK

"MiDor L'Dor" (810)

"September" (812)

"14" (814)

"KICK OFF PICNIC" (816)

"12 noon-2 pm, with Shorashim, at Mitchell park" (818)

CLICK

"20" (820)

"MiDor L'Dor begins" (822)

CLICK

"MiDor L'Dor" (810)

"December" (824)

"21" (826)

"CHANUKAH BRUNCH (10 am-12 noon, with Shorashim," (828) In the first swipe sequence, swipes 812, 814, and 818 contain date/time information, and are analyzed accordingly. Swipes 810 and 816 don't, and so serve to annotate the event.

In the second swipe sequence, the only date information is contained in swipe 820, "20". This is an incomplete portion of a date, so the month and year are carried over from the previous event. Swipe 822 serves to annotate the event.

In the third swipe sequence, swipes 824, 826, and 828 contain the date information. Swipes 810 and 828 contain other information, and so annotate the event. Note that swipe 828 contains both date and annotation information.

The third calendar type, a snack schedule, is shown in FIG. 9. This, once again, is similar to the other schedules (FIGS. 7-8); the difference here is that only one entry is relevant to the user (the date of a particular person's snack assignment). Suppose that the user wishes to enter the October 18 event. A scanning sequence to accomplish this could be:

CLICK

"October 18 Rob Rudy" (910)

"Snack Schedule" (912)

As in the previous example, one swipe 910 contains both date and annotation information. The second swipe 912 contains only annotation information. Hence, the event will be entered with a date of October 18, and with an annotation of "Rob Rudy, Snack Schedule."

A fourth calendar type, a web page, is shown in FIG. 10. This is still a schedule, but one kept primarily in the digital domain (though it can be printed in hardcopy format). Suppose that the user wishes to enter the September 6th event. A scanning sequence to accomplish this could be:

CLICK

"Game Schedule-Team B606" (1010)

"9/6 8:30" (1012)

"El Carmelo Elementary School" (1014)

The second swipe 1012 serves to completely specify the date--the other swipes 1010 and 1014 serve as annotation. Like other events, it is possible that the user may wish to add additional annotation beyond that already printed on the document--for example, in this case, directions to the school, or which child is on team "B606", and so forth. This can be either done later in the process through the system's user interface (e.g., one of the terminals 110 or 112 of FIG. 1), or, in an alternative embodiment, by allowing the scanning pen to write as well as read--the user writes additional "swipe" text himself.

As shown in FIG. 10, the name "Sam" 1016 is handwritten on the calendar. In an embodiment of the invention, a handwritten note like this one can be used to further annotate the record of the September 6 game. Preferably, the scanning pen used to input the other data on the calendar of FIG. 10 is also able to write on a hardcopy document (while simultaneously recording the physical gestures that make up the handwritten note), or to "mark up" a digital document such as a Web page. Alternatively, a previously-handwritten note can be scanned and digitized with a swipe of the scanning pen. In either case, this handwritten information, after being converted to a machine-readable format via handwriting recognition or some other means of parsing gestural symbols, is associated with the scanned record and stored in the database 310 (FIG. 3).

When the pen is used to "mark up" a digital document, it should be noted that there is no need for the pen to actually scan any information in the underlying digital document, as the document is already in the digital domain. Rather, it is sufficient for the scanning pen to indicate to the system the location of each swipe, from which the underlying information can be extracted.

The fifth calendar type, a wedding announcement, is shown in FIG. 11. Suppose that the user wishes to enter the event into the system. A scanning sequence to accomplish this could be:

CLICK

(23) "Doctor Richard Roderick Burton" (1110)

(24) "marriage" (1112)

(25) "Sunday, the ninth of November" (1114)

(26) "Nineteen Hundred and ninety-seven" (1116)

(27) "at one-thirty in the afternoon" (1118)

(28) "Saint Andrew's Episcopal Church" (1120)

(29) "13601 Saratoga Avenue" (1122)

(30) "Saratoga, California" (1124)

Swipes 1114, 1116, and 1118 specify the date and time of the event. Swipes 1110, 1112, 1122, and 1124 serve to annotate the event. The address is set forth in swipes 1120, 1122, and 1124--this information can remain part of the annotation or can be extracted by the system as described below. Note that this further information can be displayed in a hierarchical fashion, concealing details until needed. Moreover, in one embodiment of the invention, the entire announcement of FIG. 11 (or at least an additional portion thereof) is scanned and stored as an image in the database 310 (FIG. 3) in addition to the information extracted and used as an event annotation as set forth above. This approach has the advantage that additional information in the document (such as the bride's name, for example) is accessible and can be made available, if necessary, even if it is not expected to be needed at the time the key data items are extracted.

An exemplary output calendar based on information derived from FIGS. 7-11 is set forth below. As discussed above, the format of this calendar is described by an output specification 422 (FIG. 4), and can be changed dependent on the user's preferences and on the desired level of detail.

Family Calendar

     September 1997
            6         [8:30 a.m.] Game Schedule - Team B606
                      (El Carmelo Elementary School)
           14         MiDor L'Dor KICK OFF PICNIC
                      (12 noon-2 pm, with Shorashim, at Mitchell Park)
           20         MiDor L'Dor Begins
      October 1997
           18         Rob Rudy, Snack Schedule
      November 1997
            9         [1:30 p.m.] Marriage
                      Doctor Richard Roderick Burton
                      Saint Andrew's Episcopal Church
      December 1997
           21         MiDor L'Dor CHANUKAH BRUNCH
                      (10 am-12 noon, with Shorashim)

• Inventors
  Adler, Annette M.; Fishkin, Kenneth P.; Marshall, Catherine C.; Silverman, Alexander E.; Cass, Todd A.;
• Assignee
  Xerox Corporation (Stamford, CT)
• Published
  Jan-6-2004
• Current US Classes:
  707/1
  715/500
  715/530
• Application #
  219089
• International Classes
  G06F 015/00
• Field of Search
  707/530 707/531 707/500 707/1 715/530 715/531 715/500 704/9 704/10
• Examiner
  Feild; Joseph H.
• Agent
  Robb; Linda M.
• US Patent References:
  5159667
  5283818
  5528290
  5581637
  5646986
  5659639
  5692073
  5732399
  5790853
  5999664
  6032121
  6088026
  6216102
  6246410
  6249765
  6272074
  6353824
  6421670
  6424429
  6505150