Method of thematic classification of documents, themetic classification module, and search engine incorporating such a module7003519Abstract A method of thematically classifying documents, in particular for making up or updating thematic databases (42) for a search engine, includes the steps of selecting documents representative of each theme, identifying within the selected documents, elements that are characteristic of each theme, allocating a coefficient (R) to each identified element, said coefficient being representative of the relevance of said element relative to the corresponding theme, and for each document (50) for classification, identifying said elements characteristic of each theme contained in the document and, for each theme corresponding thereto, using the coefficients allocated to said elements to calculate the value of a characteristic representative of the relevance of the theme for the document (50), in order to decide whether or not the document relates to the theme. Claims What is claimed is: Description BACKGROUND OF THE INVENTION
The documents downloaded from a computer network are thus classified as a function of the themes dealt with therein, and this is done automatically. The classification method of the invention can also include one or more of the following characteristics, taken singly or in any technically feasible combination:
When searching index entries, i.e. while searching for documents that correspond to the request, it is also possible directly to access the themes associated with each element and the corresponding coefficients which are combined by multiplication in order to determine a classification of themes associated with the entire request. The invention also provides a module for thematically classifying documents, in particular for a search engine, the module being characterized in that it comprises a central processor unit having means for comparing elements extracted from each document with elements characteristic of various themes, each element being allocated a coefficient representative of the relevance of said element for a corresponding theme, and means for calculating a characteristic value representative of the relevance of a theme for the document on the basis of the coefficients of said characteristic elements that the document contains, in order to decide whether or not the document relates to said theme, said central unit being connected to means for storing documents classified by theme that can be interrogated on the basis of themes contained in a request, and in that the module has means for calculating the frequency of the element in the selected documents relating to the theme, means for calculating the frequency of the element in the selected documents that do not relate to the theme, and means for calculating the ratio between the calculated frequencies. The invention also provides a search engine for documents on a computer network, the engine comprising an indexing module for creating and updating thematic databases on the basis of documents downloaded from the computer network, and a module for interrogating thematic databases adapted to supply the references of documents corresponding to a request that has been input thereto, the search engine being characterized in that it further comprises a thematic classification module as defined above associated with the indexing module. BRIEF DESCRIPTION OF THE DRAWINGS Other characteristics and advantages appear from the following description given purely by way of example and made with reference to the accompanying drawings, in which: FIG. 1 is a flow chart showing the main operating stages of a module of the invention for thematically classifying documents for a search engine; FIG. 2 is a flow chart showing the method of calculating the elements characteristic of themes; and FIG. 3 is a flow chart showing the method of calculating the themes of a document. DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows the main stages of the method of the invention for thematically classifying documents. It is intended to enable documents downloaded from a computer network to be classified as a function of the themes they deal with. For example, it can be implemented within a search engine. Under such circumstances, it is involved in the indexing process, and also during processing of a request formulated by a user so as to determine all of the themes dealt with in the request. Nevertheless, it will be understood that other applications can be envisaged. For example, the method can be implemented at a network access point for stations using an Internet network in order to determine the nature of the web pages downloaded by the users and to filter requests in order to authorize or ban certain themes, for example themes contrary to ordre public or morals, or indeed to calculate statistics concerning uses' centers of interest. To proceed with this classification, the method comprises two distinct stages, namely: a prior first stage of acquiring the thematic vocabulary of the corpus of documents and of giving each word of the vocabulary a threshold value above which it is decided that a document containing this word relates to the corresponding theme; and also a second stage of classification proper, during which a document downloaded from the network is automatically classified as a function of the characteristic elements it contains. By way of example, this second stage takes place periodically, and only documents that have been newly created or modified are classified. The first stage of thematic vocabulary acquisition is described below with reference to FIGS. 1 to 3. As can be seen in FIG. 1, this stage starts with a manual selection step 10 from a set 12 of samples (or "corpus") of documents that are representative of each of themes A to Z used for classifying documents during the second stage. Thus, at the end of this manual selection step 10, a set of document corpuses such as 14 is available with each corpus relating to a particular theme (theme A, theme Z). Naturally, the selection step can equally well be performed by any means other than manual. During this selection step 10, a corpus 16 is also created of documents that do not relate to any of the themes A to Z, and a nomenclature 18 for the themes A to Z is defined, i.e. a list of said themes associated with subthemes relating thereto. During the following step 20, these elements are input to a thematic classification module in order to extract from each document elements that are characteristic of each theme and to give each of them a coefficient representative of its relevance relative to a corresponding theme. By way of example, this thematic classification module is in the form of a specific module of a search engine associated with an indexing module that creates or updates thematic databases. It can also be implemented in the form of a specific module provided at an access point to a computer network, in particular an Internet network. The module has software means suitable for extracting elements that are characteristic of each theme and for allocating respective coefficients representative of their relevance relative to the various themes, as described in detail below. During this step 20, the classification module extracts the elements characteristic of each theme from each of the selected documents. This extraction is performed using a computer tool of conventional type. It is therefore not described below. At the end of this step 20, lists are available of elements that are characteristic of the themes A to Z, such as the lists 22. With reference to FIG. 2, this procedure of identifying the vocabulary that is characteristic of each theme is performed successively for each element extracted from the documents in each of the corpuses 14 and 16. During a first step 24, a table of all candidate themes is cleared, i.e. a table of all themes that might correspond to an extracted element. During the following step 26, a coefficient R is calculated for each theme, where the coefficient R is representative of the relevance of the element relative to the theme. To proceed with this calculation, the frequency p of the element in the documents relating to the theme is initially calculated, and so is the frequency Q of the same element in the documents that do not relate to the theme. Thereafter the coefficient R is calculated which is constituted by the ratio of the frequencies p and q. During the following step 28, a check is made to verify that the characteristics p, q, and R lie within predetermined limits. If this is not the case, then the following element is processed. If this is the case, then the theme is added to the table of candidate themes with a score equal to the coefficient R (step 30). If any elements remain to be processed (step 32) then the procedure returns to preceding step 24. Otherwise the procedure ends. It will be observed that after the table of candidate themes has been filled it is preferably sorted by decreasing order of the scores R. It should also be observed that for each candidate theme, and up to some desired maximum number, a new element taken from the list of elements characteristic of said theme is added while remaining within the limit of a desired maximum number of the n best elements per theme selected as a function of their respective scores R. With reference once more to FIG. 1, during the following step 34, the thematic classification module proceeds by means of an appropriate algorithm automatically to calculate a threshold value corresponding to a minimum threshold to be reached in order to decide that a document containing an element characteristic of a theme does or does not relate to the theme. To perform this calculation, the classification module begins by calculating the mean value Rmean of the ratios R of the characteristic elements of each theme. Thereafter, it calculates the threshold value score-thresholdtheme using the following relationship: score-thresholdtheme=(Rmean)theme—n where theme_n designates a predetermined number which is selected to be equal to 5, for example, for most themes. It can thus be seen in FIG. 1 that after automatically calculating the scores to be reached, lists of elements that are characteristic of each of the themes A to Z are made available, such as the list 40, with each element being associated with a score to be reached, i.e. a threshold value beyond which it is considered that a document relates to the theme. After this stage of acquiring thematic vocabulary, implemented using a corpus of documents representative of various themes, the second stage of thematic classification proper can be performed in order to make up thematic databases given overall numerical reference 42 from documents 50 collected automatically from the computer network by robots such as 44. These documents 50 are input to the thematic classification module which also receives an indication of the theme nomenclature 18 and the elements available from the outcome of above-mentioned step 34. This module proceeds automatically to calculate the themes on which a document relates (step 46). To do this, it has all of the software means required for implementing the above-mentioned operations. With reference to FIG. 3, at the end of a first step 48 of this procedure, the indexing module extracts from each document 50 downloaded by the robots 44 those elements that are characteristic of the themes it contains. By way of example, this step is performed by using a hashing table to search quickly through the lists of characteristic elements for the elements contained in each document. After these elements have been extracted, the elements characteristic of the themes contained in the list 40 are identified from amongst them. For each identified element, the classification module then calculates a characteristic value representative of the relevance of each theme for the document, on the basis of the coefficients given to the element. To do this, during the following step 52, a variable "theme_score", representative of the score of the document in a given theme is set at 1, and this is done for all of the themes. Thereafter, for each element of the document, and for each theme in the tree structure of themes, if the element lies within the list of elements characteristic of the theme, then the score R is read, i.e. the value of the frequency ratio for each element, and the values read for the score R for each of the elements are multiplied together. The result of this multiplication is then used as the value for the theme_score characteristic (step 54). It is then decided that the themes recognized in document 50 are those for which the theme_score characteristic reaches or exceeds the score that is to be reached for these themes (step 56). Thus, at the end of this procedure, a set 57 of themes is available to which the downloaded document 50 relates. It will be understood that this procedure for automatically calculating the themes of documents 50 downloaded by the robots 44 enables the indexing module of a search engine to classify these documents as a function of the themes dealt with and to build up the thematic databases 42. Such a procedure for automatically calculating document themes can also be used for determining which themes are dealt with in requests made by users. To do this, starting from a request, for each of the elements of the interrogation vocabulary used in the request, the coefficients characteristic of said element relative to each of the known themes are calculated and each of these elements is associated with the coefficients and themes in such a manner that the coefficients reach a minimum value. When searching for index entries corresponding to the elements of a request, i.e. in order to calculate the results, it is thus possible to access directly the theme which is associated with the elements and also their coefficients, and these are combined by multiplication using the same procedure as that described above in order to classify the themes associated with the request as a whole. It can thus be understood that this procedure makes it possible to ask a user to refine a request, for example when the request is formulated in vague manner. It will also be understood that this procedure which enables the themes contained in a request to be identified makes it possible to monitor user requests in order to establish statistics for defining user profiles as a function of requests. It will thus be understood that the invention as described above can be used for searching for themes contained in pages downloaded from a computer network, for determining the themes contained in a request formulated by a user, and on the basis of such determination, for filtering requests and also downloaded pages in order to ban the formulation of requests or the downloading of pages relating to predetermined banned themes, and also to generate user profiles. Nevertheless, it should be observed that in the context of determining themes contained in a request, the request is considered as constituting a document input to the thematic classification module of the invention. The invention is not limited to the implementation described. In a variant, it is also possible manually to adjust the value of the threshold from which it is decided that a document does or does not bear on a given theme.
|
Same subclass Same class Consider this |
||||||||||