Category processing of query topics and electronic document content topics6182066Abstract A system for tailoring user queries and for categorizing and searching metadata about content provided on the internet and/or intranet for delivery in accordance with customized user profiles. The method and system categorizes query content and document content to facilitate the collection, storage and usage of same. Query content and document content are tokenized, vectorized, and provided for comparison processing by the inventive method. Claims Having thus described our invention, what we claim as new and desire to secure by Letters Patent is: Description FIELD OF THE INVENTION
TYPE SERVER DIRECTORY CHANNELS
Web HR /publish/benefits/401k 401k
Web HR /publish/jobopenings Jobs
Web Marketing /publish/product/specs Product Specs
Web www.badco.com /pub/productspecs Competition Specs
Web www.goodco.com /pub/products/electrnic Customer Products
PCFile engineering /projects/chipdesigns Chip Designs
PCFile marketing /reports/companalysis Competitive Anly.
FTP engineering /projects/status Status Reports
Notes engineering /specs/chipspeed A1200 Design
FIG. 2 provides a schematic illustration of the sources accessible to the Customer Intranet Server of the fictitious company, directly or through the System Server, and the channels that result from receiving or crawling those sources. Information gathered from external sources will also be mapped to the established channels, so that an end user can readily access all relevant information in a category or channel as the result of a single query. While some amount of categorization may be straightforward, such as those above-noted examples wherein any information obtained from a certain source will necessarily be provided on a given channel (i.e., with sites or site directories being mapped to the channels), the bulk of document categorization requires intensive analysis of the document contents. In addition to the crawlers which automatically funnel documents obtained from certain sources into pre-established channels, there are two other primary means by which documents are categorized. The first, and most rudimentary, is categorization by manual user interface, whereby a system administrator (or even document author) identifies the document to be loaded into the server and identifies the channels in which the document is to appear. The second, more complex, means is automatic categorization by content filtering, which is conducted by system components located at either the Customer Intranet Server or the System Server 10, the details of which are further provided below and in co-pending applications, Ser. No. 08/979,248, entitled "Method and System for Electronic Document Content or Query Content Filtering", and Ser. No. 08/980,075, entitled "Content Filtering for Electronic Documents Generated in Multiple Foreign Languages", which are assigned to the present assignee, and are being filed on even date herewith. Such automatic categorization can also be utilized at the Customer Intranet Server for the purpose of categorizing internal documents into channels, which may match or be unique from the channels provided by the System Server. Such channel definitions can be applied as well to documents received from the System Server to fill the customer-defined channels with news or other external documents. After query processing and document content categorization, it is preferable to analyze the categories to ascertain if other relationships exist among the categories, which relationships themselves may be identified as new categories or channels, which is the subject of the present invention. The foregoing co-pending applications are incorporated by reference herein, as is co-pending patent application Ser. No. 08/979,861 entitled "Method and System for Providing Access to Categorized Information from OnLine Internet and Intranet Sources," which is assigned to the present assignee. Once documents from both the internal and external sources have been categorized/assigned channels, both the documents and the assigned channels are stored in a local database at the Customer Intranet Server or associated customer location. Inventive components at the Customer Intranet Server match the channels assigned to each of the incoming documents with the user's interests as found in the user profile. Each document is then made available for access by, or is sent to, the user whose interests it matches. The System Server's above-noted functions may be provided as part of a customer intranet, wholly outside of the customer domain, or divided in function between the two locations. In the "outside" example, all document collection and categorization would be done at the System Server as a service of the provider. Documents found on the external internet, as well as those which may be supplied from the customer's own intranet and/or databases, would be analyzed and categorized at the provider location. In the instance where the customer wishes to additionally be a provider to end users, two alternative scenarios are possible. Under the first scenario, an outside provider would still assemble and categorize documents from outside sources and make them available at the customer's server. The customer's server would also be adapted to perform assembly and categorization of "in-house" documents, merging of the in-house assemblage with the categorized documents from outside sources, matching the resultant merged documents to user request profiles, and disseminating the matching results to the user. The second alternative implementation would locate all categorization functionality at the customer location. In all three implementations, the customer location would retain the capability for receipt of user request input, creation and storage of the user profile, matching of the user profile to the categories or channels into which the documents are placed, and provision of the matched documents for end user review. The customer site is provided with the capability for building applications to create a series of different user interfaces with different interaction means, different restrictions for user access (e.g., providing some users access to only documents from outside sources, while others would have access to both externally-obtained and internally-generated documents), and different levels of query and content complexity. For the detailed descriptions of the processing "stages," including user query analysis and profile creation, document categorization, and matching, it is to be noted that the same types of analyses can frequently be applied at each stage. For example, finding relationships between two seemingly disparate user query subject categories can parallel the effort to identify commonality of subject matter from two input documents, as well as a subsequent effort to match the profile to a category/channel. Therefore, where appropriate, the ensuing processes will reference one, two or all of profile analysis, document content categorization, and matching stages. Users of the system initially specify which topics are of interest. This specification takes the form of a simple subscription to pre-defined user categories, a modified subscription whereby the user can alter or add to the pre-defined user categories, a user-customized set of queries, or any combination of the foregoing. Each query represents a topic, and can additionally contain boolean, fuzzy, proximity and/or hierarchical operators. A set of topics preferred by a user is know as a user profile. The present method reduces each query to one or more vector entries with the entry's index into the vector corresponding to a hash of the query's textual expression of the importance of that query to the overall topic/profile. A query can be either a single token (word or phrase) or a combination of tokens which includes boolean, fuzzy, proximity and/or hierarchical operators. Token IDs are assigned to each query item as hereinafter detailed. Automatic query processing, as well as document content categorization, is optimized in the present invention by first tokenizing the content thereof. In such a tokenization process, all the word/phrases are first identified as units, then stemmed. After all stop words and phrases are filtered out, only a few of the original word/phrases are left. These surviving words/phrases are called tokens. The tokens are usually just the stems of the original words, or made-up labels which correspond to phrases. The stems or made-up labels are referred to as "terms". Terms are strings, and since the system must handle quite a few thousand terms, the total memory which can be consumed by terms could take up a significant amount of computer memory. Therefore, a hash function is provided to assign unique token IDs to the terms (which may also consist of expressions containing words and phrases as terms combined with a variety of query operations) found in the documents and queries. The term strings are replaced by 32 bit integers. A "reverse dictionary" can be maintained which comprises a lexicon with token IDs as the keys and the words, phrases, queries as the values. However, if the need is to mark the document with categories, and not to catalog and retrieve based on the specific tokens matched, a lexicon will not be needed. Clearly, when comparisons are being made, comparisons of 32 bit integers will be significantly faster than the prior art string comparisons. Textual messages are likewise mapped to vectors using the same procedures as were used for the topics, above. All vectors are then normalized. Classification and matching are thereby reduced to vector processing. Query processing suffers from the drawback that, even with tokenizing and vectorizing, a great deal of redundancy may be contained in large query sets. The redundancy increases CPU and memory consumption requirements for any of the categorization processes based on the query sets. Query processing can be streamlined by recognizing possible hierarchical relations between queries in a set that has been previously indexed and vectorized, some of which may correspond to known topic categories or channels. In order to streamline the query processing, after vectorization, the following steps are implemented: First, one calculates the cosine measure (dot product) of every query vector against every other vector. This will provide a similarity measure of every query against every other query in the database. The system stores all similarity measures that equal or exceed a pre-set threshold in a sparse matrix. Those query vectors having similarity measures with scores below the threshold are assumed to have nothing in common, and therefore, are assigned an implicit similarity measure equal to zero. Standard clustering methods are applied to the sparse matrix of similarity measures. Applying a second threshold, the clusters are divided into two groups comprising (a) clusters of vectors whose similarity measures exceeds or equals the second threshold; and (b) clusters of vectors whose similarity measures do not exceed the second threshold. Membership in group (a) or (b) is determined by comparison to a predefined similarity threshold say, for example, 60%. Thus, those queries in a cluster that share greater than or equal to 60% of their tokens belong to group (a), while those that don't belong to group (b). The differences between groups (a) and (b) is that the queries in (b) are not as strongly related as those in (a). The query vectors in group (a) share most of their terms. When shown a cluster of such queries, the information analyst must ask the following questions: "Are these queries related to one another?"; "If they are related, are they part of the same branch or related branches in the topic hierarchy?"; "If they are not currently related to one another, should they be related?"; "If so, what is the best way to relate them?"; and, "If they should not be related, what is the best way to avoid this clustering (overlap) of queries and of discriminating between them in the future?". The queries in groups (a) and (b) may indicate to the information analyst: new links between previously unrelated pre-existing categories (forming new hierarchy branches); a strengthening in the links between previously related categories (consolidation and strengthening of existing hierarchy branches); or, new links between pre-existing categories and entirely new categories (again, forming new hierarchy branches). If any bogus links are discovered between totally unrelated queries, those links must be avoided by refining/enhancing the queries. Using the results of the clustering process, the information analyst can be presented with a list of queries in each cluster in group (a) or (b) and decide whether the queries truly have anything in common or not. If the queries already belong in branches of the same hierarchy, their tags will make this fact obvious, and the analyst may skip further analysis. If, on the other hand, the tags do not show any relationship between the queries, the analyst may decide that further analysis of the individual queries is required. Finally, if two or more queries in a cluster have quite a few things in common, the common terms can be made into a single query vector and this query vector can be replaced by a single term in all of the original queries. This single replacement term corresponds to and represents the new query vector. The foregoing procedure will reduce the amount of redundancy in the system. Of course, the information analyst must first decide that the terms involved are truly common to all the queries and that those terms will likely remain common throughout the life of the queries before consolidating them into a common query vector. In a similar manner, vector clustering can be utilized to automatically find new topic categories among the document content categories. Again assuming that all documents have been pre-indexed and converted into normalized vectors, one calculates the cosine measure (dot product) of every document vector against every other document vector. This will provide a similarity measure of every document against every other document in the database. The system stores those similarity measures that equal or exceed a preset threshold in a sparse matrix. Those documents vectors having similarity measures with scores below the threshold are assumed to have nothing in common, and therefore are assigned an implicit similarity measure equal to zero. Once again, standard clustering methods are applied to the sparse matrix of similarity measures. Each cluster produced will fall into one of the following groups: (A) document vectors in the cluster mostly share common pre-existing category tags; (B) document vectors in the cluster share some common pre-existing tag categories; (C) document vectors in the cluster share no category tags. The documents in group (A) closely match well known, pre-existing categories. Thus, at first sight, they hold little interest to information analysts. But, on further analysis, the analyst may find that much can be learned from this group. For example, the analyst may ask: how closely related are these pre-existing categories that they show up in almost every document in the group? The document in groups (A) and (B) may indicate: new links between previously unrelated pre-existing categories; a strengthening in the links between previously related categories; or, new links between pre-existing categories and entirely new categories. The documents in group (C) indicate the existence of previously unknown categories and of links between them. This is the most important category for the information analyst. For each cluster found, and for each group of documents of type (A) and (B) within that cluster, and for each matched category within a group, a summary vector is calculated from all the document vectors in the group matching that category. A summary vector is a single vector that best represents a cluster of neighboring document vectors. It represents the average vector in the cluster and is calculated by taking the centroid of all of the vectors in the cluster. This summary vector is refined by a process of comparison with other summary vectors. The final, refined summary vector represents the query which will retrieve those document with the highest recall and precision possible when issued against the corpus of documents stored in the document database. This new query vector is then compared to the one associated with the original category, to determine to what extent they match. After this process, if there is enough of an overlap, an improved version of the original query vector associated with the category is produced. If, on the other hand, they only partially overlap, a new category and query is produced together with an improved version of the original query vector associated with the original category. The new link between them is implicit in the terms which both query vectors share. For each cluster found, and for each group of document of type (C) within that cluster, a summary vector is calculated from all the document vectors in the group. This summary vector is refined by the process of comparison with other summary vectors. The final, refined summary vector represents the query which will retrieve those documents with the highest recall and precision possible when issued against the corpus of documents stored in the document database. This query also represents a new category. Every new category requires a new tag/label that best represents that category. This tag is put together initially by concatenating the most representative (i.e., highest score) terms of the query associated with the category. A quick scan of the text of documents retrieved by the query will locate those representative terms in their original context, will determine if those terms are part of any collocations which are not part of any term yet, and, if so, will replace those terms in the label with their most common collocations. The final label is scanned by a small parser specializing in noun and verb phrases as may appear in a category label, to make sure that it is syntactically correct. Refined categories have their labels/tags enhanced by a process identical to the one described immediately above. The invention has been described with reference to several specific embodiments. One having skill in the relevant art will recognize that modifications may be made without departing from the spirit and scope of the invention as set forth in the appended claims.
|
Same subclass Same class Consider this |
||||||||||