Apparatus and a method for retrieving image objects based on correlation with natural language sentence parameters

Abstract

An apparatus for retrieving image objects in accordance with a natural language sentence according to the present invention includes: an input section for receiving a natural language sentence; a language processing section for parsing the natural language sentence by referring to a dictionary for language analysis so as to obtain a syntactic structure of the natural language sentence; a situation element division section for converting the syntactic structure of the natural language sentence into a semantic structure of the natural language sentence and for dividing a situation represented by the semantic structure of the natural language sentence into at least one situation element by referring to a situation element division knowledge base; an image database for storing at least one image object corresponding to the at least one situation element; and a retrieval section for retrieving at least one image object from the image database by using the situation element as an image retrieval key.

Claims

What is claimed is:

1. An apparatus for retrieving image objects in accordance with a natural language sentence, comprising:

an image database for storing a plurality of image objects, each of the plurality of the image objects comprising image attribute data selectively representative of a meaning of a single region and a combination of a plurality of regions making up a portion of an image;

input means for receiving a natural language sentence;

a knowledge base for defining a semantic structure of the natural language sentence;

extracting means for extracting at least one predicate and at least one case element from the natural language sentence;

judging means for judging whether the extracted predicate and the extracted case element are correlated with each other by referring to the knowledge base;

correlating means for correlating the extracted predicate and the extracted case element with each other in accordance with the judging result, so as to define an image retrieval key representing a partial meaning of the natural language sentence; and

retrieval means for retrieving at least one image object from the plurality of image objects stored in the image database based on a semantic similarity between the image retrieval key and the image attribute data, the retrieval means being selectively operative based on the semantic similarity to retrieve image objects representative of only portions of images.

2. An apparatus according to claim 1, wherein the image attribute data includes region data defining one of a single region and a combination of a plurality of regions included in the image and information representing the meaning of one of the single region and the combination of the plurality of regions.

3. An apparatus according to claim 2, wherein the information is represented by the natural language sentence.

4. An apparatus according to claim 1, wherein the retrieval means comprises means for providing region availability information defining whether the image attribute data is available or not available in accordance with the semantic similarity between the image retrieval key and the image attribute data.

5. An apparatus according to claim 4, wherein the apparatus further comprises display means for displaying at least a portion of the image object retrieved by the retrieval means in accordance with the region availability information.

6. An apparatus according to claim 1, wherein the knowledge base comprises a plurality of case frames for defining a meaning of the natural language sentence, and wherein each of the plurality of the case frames comprises:

a word defining a predicate in the natural language sentence;

a slot describing a name of a case element related to the word and a constraint for the case element to satisfy; and

situation element data describing knowledge for correlating a predicate with at least one case element.

7. An apparatus according to claim 1, wherein the correlating means classifies the image retrieval key into a group related to a background and a group related to a foreground based on the situation element division data of the case frames.

8. An apparatus for retrieving image objects in accordance with a natural language sentence, comprising:

an image database for storing a plurality of image objects, each of the plurality of image objects comprising image attribute data selectively representative of a meaning of a single region and a combination of a plurality of regions making up a portion of an image;

input means for receiving a natural language sentence;

parsing means for parsing the natural language sentence so as to obtain an analysis frame representing a meaning of the natural language sentence, the analysis frame including at least one element;

correlating means for correlating the at least one element of the analysis frame with one another so as to define an image retrieval key representing a partial meaning of the natural language sentence; and retrieval means for retrieving at least one image object from the plurality of image objects stored in the image database based on a semantic similarity between the image retrieval key and the image attribute data, the retrieval means being selectively operative based on the semantic similarity to retrieve image objects representative of only portions of images.

9. An apparatus according to claim 8, wherein the analysis frame includes a predicate, a case element related to the predicate, and a value of the case element.

10. An apparatus according to claim 8, wherein the correlating means defines either an image retrieval key related to a background or an image retrieval key related to a foreground in accordance with the partial meaning of the natural language sentence.

11. A method for retrieving image objects from an image database in accordance with a natural language sentence, the image database storing a plurality of image objects, each of the plurality of image objects including image attribute data selectively representative of a meaning of a single region and a combination of a plurality of regions making up a portion of an image, wherein the method comprises the steps of:

a) receiving a natural language sentence;

b) parsing the natural language sentence so as to obtain an analysis frame representing a meaning of the natural language sentence, the analysis frame including at least one element;

c) correlating the at least one element of the analysis frame with one another so as to define an image retrieval key representing a partial meaning of the natural language sentence; and

d) retrieving at least one image object from the plurality of image objects stored in the image database based on a semantic similarity between the image retrieval key and the image attribute data, such retrieving being selectively operative based on the semantic similarity to retrieve image objects representative of only portions of images.

12. A method according to claim 11, wherein the analysis frame includes a predicate, a case element related to the predicate, and a value of the case element.

13. A method according to claim 11, wherein the step c) comprises a step for defining either an image retrieval key related to a background or an image retrieval key related to a foreground in accordance with the partial meaning of the natural language sentence.

Description

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an apparatus and a method for retrieving image objects stored in an image database by using a natural language. In particular, the present invention relates to an apparatus and a method for easily retrieving images that are related to the content of a document, such as a pamphlet or a manual, created by using a document creation device such as a wordprocessor or a DTP (Desk Top Publishing) device.

2. Description of the Related Art

Since an image can carry a great deal of information which is concrete relative to that carried by words, images are used in many aspects of our daily life, e.g., pamphlets, magazines, TV news, etc. Therefore, there has been an increasing demand for a system that allows any person to easily retrieve a desired image from a database in which a large amount of images are stored. Conventional methods for retrieving images include the following:

(1) Methods in which keywords are used:

According to methods in this category, several keywords are previously assigned to each image stored in a database. When a user inputs several keywords related to a desired image, the similarity between the input keywords and the keywords assigned to each image is calculated by using the distances between keyword vectors, for example, so that images are presented to the user in descending order of similarity. The use of keyword vectors is disclosed in SHIBATA, INOUE, "Associative Retrieval Method for Image Database", THE THESIS JOURNAL OF THE INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS, D-II, Vol. J73-D-II, No. 4, pp. 526-534, 1990, for example.

(2) Methods in which similar images are traced:

According to methods in this category, several images which will serve as clues for retrieval are displayed for a user. As in the methods described in (1) above, several keywords are previously assigned to each image stored in a database. When the user selects one that seems most similar to a desired image out of the group of displayed images, the apparatus calculates images that are similar to the selected image by using the distances between keyword vectors as in (1) above, for example, and displays images similar to the selected image in descending order of similarity. The user arrives at the desired image through a repetition of this cycle.

(3) Methods in which hand-drawn sketches are used:

According to methods in this category, a user draws a sketch by hand or with the use of a drawing tool, etc. to the best of his/her memory. Then, the similarity between the sketch drawn by the user and each stored image is calculated based on the degree of coincidence of the color of each pixel, so as to present images in descending order of similarity.

Hereinafter, the features of the above methods (1) to (3) will be described.

The method of (1), in which keywords are used, has the following advantages, for example: any keyword can be used for the retrieval as long as it is previously registered in the apparatus; it is easy to newly register a keyword that has not been registered in the dictionary of the apparatus; and relatively decent retrieval can be conducted by simply correlating the keywords so as to be superior or subordinate to one another. Therefore, this method is frequently used for a document database.

However, this method may not necessary allow a user to obtain the desired retrieval results when the method is applied to the retrieval of images. Specifically, keywords alone cannot quantitatively express the meaning of a given image (e.g., the situation of the image expressed in the form of one or more sentences), so that a given keyword, which is assigned by one who registers images in a subjective manner, may not always be the best keyword for a user.

With the method of (2), which utilizes similar images, a desired image can be retrieved while a user browses through presented images, which makes it easier for the user to see the development of the retrieval, as compared with the method of (1), where keywords are used.

However, this method has problems similar to method (1). Specifically, the apparatus calculates the similarity between one image that is selected by a user and all the images stored therein by using keywords that are previously assigned to the stored images, so that the user may not necessarily be able to obtain the desired retrieval results.

Thus, both of the methods of (1) and (2) have a problem in that, while the contents of the images are expressed with the use of keywords, the correlation between the keywords and the images may vary depending on the user, so that a user may not necessarily be able to obtain the desired retrieval results.

The method of (3) above, in which hand-drawn sketches are used, is intended to solve the abovementioned problem by presenting an image to the apparatus. According to this method, a user draws a sketch of a desired image by using a drawing processor that utilizes a mouse. The sketch may include line drawings of two digits (of 1 or 0), and also color information. The sketch is immediately subjected to a data conversion process so as to calculate the similarity between the sketch and the stored images. Then, similarity is calculated by using the edge information, color information, etc. of the sketch. As a result, images which are similar to the input sketch are sequentially presented in descending order of similarity.

The above method has an advantage in that, since previously quantified information, such as contour information and color information of images, is used for the calculation of similarity, no mental work such as searching for inquiry keywords in a dictionary is required, so that the user only needs to know the content of the desired image to conduct the retrieval.

However, the above method has a problem in that the user cannot draw a sketch when he/she does not know the content of a desired image, so that he/she may not always be able to retrieve the desired image. Moreover, since contours, colors, etc. of images are used as information during the calculation of the similarity, the calculation cost of the similarity becomes large, resulting in a large amount of time required for the retrieval, especially in the case of a large scale database.

Furthermore, in any of the above methods (1) to (3), it is generally the case that, when a desired image is not present in the database, the image is obtained through an edition/synthesis process of a plurality of images. Thus, it is required to first retrieve every image that contains objects necessary for the synthesis of the desired image, and then cut out the necessary object portions and combine the cut-out object portions with a background image so as to obtain the desired image. This process requires a large amount of effort on the part of the user.

An exemplary image retrieval method in which the above methods (1) (where keywords are used) and (2) (where similar images are traced) are combined is disclosed in INOUE, SHIBATA, NAKASU, "Pictorial Parts Filing System for Image Synthesis", THE THESIS JOURNAL OF THE INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS, Vol. J72-D-II, NO. 11, pp. 1824-1832, November 1989. An example of the image retrieval method (3) is disclosed in Japanese Laid-Open Patent Publication No. 2-51775.

Such conventional image retrieval techniques have a problem in that, in the case where keywords are used for the inquiry, the retrieval results may not necessarily reflect what a user desires to retrieve because of being unable to express the meaning of the image in a quantitative manner; or, in the case where sketches are used for the inquiry, retrieval of the image is impossible if the user does not know what the desired image is like, even if provided with a means to accurately inform the apparatus of what the user desires to retrieve.

In addition, when the desired image is not present in the image database, the user of a conventional image retrieval apparatus is required to create the desired image through an edition/synthesis process. This edition/synthesis process must be performed by manual labor, which requires a large amount of effort on the part of the user.

SUMMARY OF THE INVENTION

An apparatus for retrieving image objects in accordance with a natural language sentence according to the present invention includes: input means for receiving a natural language sentence; language processing means for parsing the natural language sentence by referring to a dictionary for language analysis so as to obtain a syntactic structure of the natural language sentence; situation element division means for converting the syntactic structure of the natural language sentence into a semantic structure of the natural language sentence and for dividing a situation represented by the semantic structure of the natural language sentence into at least one situation element by referring to a situation element division knowledge base; an image database for storing at least one image object corresponding to the at least one situation element; and retrieval means for retrieving at least one image object from the image database by using the situation element as an image retrieval key.

In one embodiment of the invention, each of the image objects includes image attribute data relating to at least one region included in an image, and the retrieval means subjects the image objects obtained from the image database in accordance with the image attribute data to a further screening.

In another embodiment of the invention, the image attribute data includes region data defining at least one region included in the image and information representing the meaning of the at least one region.

In still another embodiment of the invention, the information is represented by the natural language sentence.

In still another embodiment of the invention, the retrieval means includes means for calculating a similarity between each of the image retrieval key and each of the image attribute data, and means for providing region availability information defining whether the image attribute data is available or not available in accordance with the similarity.

In still another embodiment of the invention, the apparatus further includes display means for displaying at least a portion of the image object retrieved by the retrieval means in accordance with the region availability information.

In still another embodiment of the invention, the situation element division knowledge base includes a plurality of case frames for defining a meaning of the natural language sentence, and wherein each of the plurality of the case frames includes: a word defining a predicate in the natural language sentence; a slot describing a pair consisting of a name of a case element related to the word and a constraint for the case element to satisfy; and situation element data describing knowledge for dividing the situation into the at least one situation element.

In still another embodiment of the invention, the situation element division means classifies the at least one situation element into a group related to a background and a group related to a foreground based on the situation element division data of the case frames.

In still another embodiment of the invention, the dictionary for language analysis includes a rule for deleting words that are unnecessary for retrieving images from the natural language sentence, and the language processing means includes image retrieval preprocessing means for deleting words from the natural language sentence in accordance with the rule.

Alternatively, an apparatus for retrieving image objects in accordance with a natural language sentence according to the present invention includes: an image database for storing a plurality of image objects, each of the plurality of image objects including image attribute data representing a meaning of a region included in an image and pixel data corresponding to the region; input means for receiving a natural language sentence; parsing means for parsing the natural language sentence so as to obtain an analysis frame representing a meaning of the natural language sentence, the analysis frame including at least one element; correlating means for correlating the at least one element of the analysis frame with one another so as to define an image retrieval key representing a partial meaning of the natural language sentence; and retrieval means for retrieving at least one image object from the image database based on a semantic similarity between the image retrieval key and the image attribute data.

In one embodiment of the invention, the analysis frame includes a predicate, a case element related to the predicate, and a value of the case element.

In another embodiment of the invention, the correlating means defines either an image retrieval key related to a background or an image retrieval key related to a foreground in accordance with the partial meaning of the natural language sentence.

In another aspect of the present invention, a method for retrieving image objects from an image data-base in accordance with a natural language sentence is provided, the image database storing a plurality of image objects, each of the plurality of image objects including image attribute data representing a meaning of a region included in an image and pixel data corresponding to the region, wherein the method includes the steps of: a) receiving a natural language sentence; b) parsing the natural language sentence so as to obtain an analysis frame representing a meaning of the natural language sentence, the analysis frame including at least one element; c) correlating the at least one element of the analysis frame with one another so as to define an image retrieval key representing a partial meaning of the natural language sentence; and d) retrieving at least one image object from the image database based on a semantic similarity between the image retrieval key and the image attribute data.

In one embodiment of the invention, the analysis frame includes a predicate, a case element related to the predicate, and a value of the case element.

In another embodiment of the invention, the step c) includes a step for defining either an image retrieval key related to a background or an image retrieval key related to a foreground in accordance with the partial meaning of the natural language sentence.

According to the present invention, when registering image objects in an image database, region data indicating regions defining constituent elements of the images is created, and parts of regions and image content information are stated as image attribute data of image objects.

In accordance with the image retrieval apparatus of the present invention, a natural language sentence representing the content of a desired image is input via an input section. A language processing section parses, i.e., analyzes the syntactic structure of, the input natural language sentence. A situation element division section converts the syntactic structure of the natural language sentence into a semantic structure. In other words, a situation represented by the input natural language sentence is divided into partial situation elements constituting the situation, and image retrieval keys corresponding to the respective divided situation elements are generated. Herein, a situation element is defined as the minimum sentence unit that describes a situation.

In an image retrieval section, image objects related to every one of the divided situation elements are retrieved. The retrieved image objects may be subjected to a further screening where image object candidates are further narrowed down.

When the semantic structure of the input natural language sentence and the semantic structure of the retrieval image object completely coincide with each other, it means that the desired image object has been found. Accordingly, a display section displays the retrieved image object. When the semantic structure of the input natural language sentence and the semantic structure of the retrieval image object do not completely coincide with each other, the display section displays image objects corresponding to a foreground and image objects corresponding to a background, both of which are necessary for expressing the semantic structure of the input natural language sentences in a separate manner. Moreover, if necessary, it is possible to edit the displayed image objects.

Thus, the invention described herein makes possible the advantage of providing an image retrieval apparatus which makes it possible to accurately inform the apparatus of what a user desires to retrieve even if the user does not know what the desired image is like, and ensures that the retrieval results will reflect what the user desires to retrieve even when the desired image is not present in the image database, and that the desired image can be easily arrived at through edition/synthesis.

This and other advantages of the present invention will become apparent to those skilled in the art upon reading and understanding the following detailed description with reference to the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a diagram showing an example of an image to be stored in an image database.

FIG. 1B is a diagram showing examples of regions included in the image shown in FIG. 1A.

FIG. 1C is a diagram showing the structure of image attribute data corresponding to the image shown in FIG. 1A.

FIG. 2 is a block diagram showing a configuration for an image retrieval apparatus according to an example of the present invention.

FIG. 3 is a diagram showing the structure of an image object retrieval dictionary.

FIG. 4 is a diagram showing an exemplary analysis result of a syntactic structure expressed in a tree structure.

FIG. 5 is a diagram showing the structure of a case frame included in a situation element division knowledge base.

FIG. 6 is a diagram showing the structure of a case frame representing a semantic structure of a natural language sentence.

FIG. 7 is a flowchart showing the procedure of an image retrieval process according to the present invention.

FIG. 8 is a flowchart showing the procedure of an IMAGE OBJECT RETRIEVAL ROUTINE in the image retrieval process according to the present invention.

FIG. 9 is a flowchart showing the procedure of an IMAGE OBJECT SELECTION ROUTINE in the image retrieval process according to the present invention.

FIG. 10 is a flowchart showing the procedure of a process by a display section.

FIG. 11A is a diagram showing an example of an image to be stored in an image database.

FIG. 11B is a diagram showing examples of regions included in the image shown in FIG. 11A.

FIG. 11C is a diagram showing the structure of image attribute data corresponding to the image shown in FIG. 11A.

FIG. 12 is a diagram showing the structure of a case frame representing a semantic structure of a natural language sentence.

FIG. 13 is a diagram showing a result of the edition of retrieved image objects.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, the present invention will be described with reference to the accompanying figures.

FIG. 1A Shows an exemplary image to be stored in an image database. This image represents a scene where "Mike is coaching John riding a bicycle in .largecircle..largecircle. Park, where there are trees growing". As shown by this example, an image represents a specific meaning. Moreover, individual regions that are included in an image can be interpreted as representing meanings that may be different from the meaning of the entire image. For example, if a scene where "Mike is coaching John riding a bicycle" is interpreted with respect to the individual regions included in the image, an interpretation that "Mike is riding a bicycle" and an interpretation that "John is running" can be obtained. Furthermore, if a scene of "roadside trees" is interpreted with respect to the individual regions included in the image, an interpretation can be obtained that there are three "trees".

The present invention provides an apparatus for conducting a retrieval of an image from an image database based on the meaning of the whole or part of such an image. An image which is stored in an image database can be used for the synthesis of other images. In this respect, each image stored in an image database is referred to as an "image object" in the present specification. As used herein, an image object includes pixel data, region data, and image attribute data. The pixel data consists of a set of pixel values. For example, in the case of an image object of 640.times.480 dots, the pixel data consists of a set of 640.times.480 pixel values. The region data is data for specifying the position of at least one region included in a given image object. For example, the region data may be a set of coordinates defining the contour of a region. The pixel data and the region data can be stored in the image database by a known method. The image attribute data defines the relationship between the meanings of the pixel data and the respective regions. The meaning(s) represented by the pixel data may correspond to one region or a plurality of regions.

FIG. 1B shows exemplary regions included in the image of FIG. 1A. As shown in FIG. 1B, the image of FIG. 1A includes regions X.sub.1 to X.sub.5. As mentioned above, the region data specifies the positions of regions X.sub.1 to X.sub.5.

FIG. 1C shows the structure of image attribute data corresponding to the image of FIG. 1A. The image attribute data includes information 101, which represents the meaning of the whole or part of the pixel data, and a region section 102 related to the whole or part of the pixel data. The information 101 may, for example, be described by a natural language sentence. The natural language sentence may be replaced by a sentence in any other language that permits retrieval or searching. For example, the meaning of the whole or part of the pixel data may be expressed in the form of a sequence of words of a natural language.

In FIG. 1C, image attribute data 103 defines that the meaning of the pixel data that corresponds to region X.sub.1 is that "John is running". Image attribute data 104 defines that the meaning of the pixel data thatcorresponds to region X.sub.2 is that "Mike is riding a bicycle". The image attribute data 103 and 104 each illustrate an example where the meaning of one set of pixel data corresponds to one region. However, the meaning of one set of pixel data may correspond to two or more regions. For example, image attribute data 105 defines that the meaning of the pixel data that corresponds to a region consisting of regions X.sub.1 and X.sub.2 is that "Mike is coaching John riding a bicycle". Image attribute data 106 defines that the meaning of the pixel data that corresponds to a region consisting of regions X.sub.3 to X.sub.5 is "roadside trees". Thus, the image attribute data is defined in such a way that the meaning of pixel data corresponding to a combination of a plurality of regions is different from the meaning of any pixel data corresponding to one region.

In the pixel attribute data 107, no corresponding region is defined for the meaning of the pixel data. In other words, the pixel attribute data 107 adopts an expression in which the corresponding region for the meaning of the pixel data is omitted. Such an omission indicates that the meaning of the pixel data is related to the entire image shown in FIG. 1A, including regions X.sub.1 to X.sub.5 shown in FIG. 1B (i.e., the background region of the image). Accordingly, the image attribute data 107 defines that the meaning of the pixel data corresponding to the background region is ".largecircle..largecircle. Park". Thus, it will be seen that, in the case where a region is defined to correspond to the meaning of given pixel data in pixel attribute data, the pixel data relates to the foreground region of the image, and that, in the case where no region is defined to correspond to the meaning of given pixel data in pixel attribute data, the pixel data relates to the background region of the image. Accordingly, it is possible to determine whether the meaning of given pixel data in pixel attribute data relates to the foreground region or the background region of an image by checking whether or not any regions are defined to correspond to the pixel data.

FIG. 2 shows a configuration for an image retrieval apparatus according to an example of the present invention. As shown in FIG. 2, the image retrieval apparatus includes: an input section 201 for inputting natural language sentences; a language processing section 202 for analyzing the syntactic structure of a natural language sentence by referring to a dictionary 206 for language analysis; a situation element division section 205 for converting the syntactic structure of the natural language sentence into a semantic structure by referring to a situation element division knowledge base 210, and for dividing a situation represented by the semantic structure of the natural language sentence into at least one situation element so as to generate an image retrieval key corresponding to the situation element; an image retrieval section 211 for retrieving image objects from an image database 212 based on the image retrieval key; and a display section 215 for displaying the retrieved image objects and for, if necessary, editing the image objects.

The image database 212 includes an image object retrieval dictionary 213 for retrieving image objects and an image object database 214 for storing image objects.

FIG. 3 shows a structure of the image object retrieval dictionary 213. The image object retrieval dictionary 213 includes a plurality of image object retrieval data. Each image object retrieval data in FIG. 3 corresponds to one line of the image object retrieval dictionary 213. Each image object retrieval data includes a keyword 301 indicating the content of an image object, a pointer 302 to other image object retrieval data that are related to the image object retrieval data itself, and a pointer 303 to image objects that are related to the keyword 301. As shown in FIG. 3, the keyword 301, the pointer 302, and the pointer 303 are partitioned by double vertical lines.

As an example of image object retrieval data, image object retrieval data 304 shown in the first row of FIG. 3 will be described below. The image object retrieval data 304 has "John" as the keyword 301, at least "human (Upper Class)" as the pointer 302, and at least "Image 1" and "Image 2" as the pointer 303. The value "human (Upper Class)" indicates that any image object retrieval data having "human" as the keyword 301 is correlated as "Upper Class" to the image object retrieval data 304. The values "Image 1" and "Image 2" represent pointers to the image objects related to "John" by sequential numbers assigned to such image objects. Specifically, the values "Image 1" and "Image 2" indicate that image 1 and image 2 are stored in the image object database 214 as image objects related to "John".

With reference to FIG. 2, the language processing section 202 analyzes the syntactic structure of an input natural language sentence by referring to the dictionary 206 for language analysis. The dictionary 206 for language analysis includes an image-retrieval proprocessing knowledge base 207, a word dictionary 208 and a grammar dictionary 209. The language processing section 202 includes: an image-retrieval preprocessing section 203 for, if necessary, deleting unnecessary words and/or correcting words among those included in the input natural language sentence by referring to the image-retrieval preprocessing knowledge base 207; and a parsing section 204 for analyzing the syntactic structure of the natural language sentence as processed by the image-retrieval preprocessing section 203 by referring to the word dictionary 208 and the grammar dictionary 209. The method for parsing by the parsing section 204 is a technique known in the field of natural language processing. For example, a method for implamenting the parsing section 204, the word dictionary 208, and the grammar dictionary 209 is disclosed in "Basic Technique of Natural Language Processing", pp. 1 to 23, THE INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS, 1988. The parsing section 204 of the image retrieval apparatus according to the present invention may be embodied by using any parsing method as long as the parsing results of the input natural language sentences are properly output from the parsing section 204.

Hereinafter, the process performed at the language processing section 202 will be briefly described with respect to a case where The following natural language sentence (1) is input to the image retrieval apparatus by the input section 201:

"Tom is running with John in the first group at XX Marathon.": (1)

The image-retrieval preprocessing section 203 deletes unnecessary words and/or corrects any words, from the input natural language sentence (1), that are unnecessary for expressing an image. For example, the image-retrieval preprocessing section 203 performs image-retrieval preprocessing such as deletion of prefixes, words expressing mood information and/or honorific expressions, and conversion of any conjugated/declined words into the base forms thereof. The image-retrieval preprocessing is performed by referring to image-retrieval preprocessing knowledge stored in the image-retrieval preprocessing knowledge base 207.

The image-retrieval preprocessing knowledge includes a rule that "If a BE verb or a conjugate of the BE verb is followed by a present participle of a verb, then the BE verb or the conjugate of the BE verb is removed, and the present participle of the verb is converted into the base of the verb", for example. The above rule defines that, if the input natural language sentence includes mood information, such as the progressive form, any portion expressing the progressive form should be removed and converted into the base form. For example, in the natural language sentence (1), the second word "is" (BE verb) is followed by the third word "running" (present participle of the verb "run"), so that the above rule is applied. As a result, the second word "is" is removed, and the third word "running" is converted into "run" (base form). In other words, the natural language sentence (1) is converted into the following natural language sentence (1'):

"Tom run with John in the first group at XX Marathon.": (1')

By performing the above process with the image-retrieval preprocessing section 203, it becomes unnecessary to include grammar rules of conjugations, grammar rules encompassing prefixes, and the like in the grammar dictionary 209. Thus, the number of grammar rules to be included in the grammar dictionary 209 can be reduced. As a result, the processing speed of the parsing section 204 can be increased. However, the image retrieval preprocessing section 203 may be omitted. The reason is that, in some cases, it may make for a more accurate retrieval to take into account conjugation and the like.

The image-retrieval preprocessing knowledge is incorporated in the image retrieval apparatus by using a programming language. For example, if the programing language is a procedural language, the image-retrieval preprocessing knowledge can be easily expressed by using if-then type condition branches. The above-mentioned rule may be expressed as follows by using a procedural language:

    ______________________________________
    if (a BE verb or a conjugate of the BE verb is
    followed by a present participle of a verb)
    then {remove the BE verbh or the conjugate of the
    BE verb.
    convert the present participle of the verb
    into the base form of the verb.
    ______________________________________

    ______________________________________
    if (condition for performing an image-retrieval
           preprocess)
    then {
           perform the image-retrieval preprocess.
    .
    .
    .
    if (condition for performing an image-retrieval
           preprocess)
    then {
           perform the image-retrieval preprocess.
    }
    ______________________________________

• Inventors
  Okamoto, Shusaku;
• Assignee
  Matsushita Electric Industrial Co., Ltd. (Kadoma, JP)
• Published
  Nov-4-1997
• Current US Classes:
  382/305
  704/9
  707/3
• Application #
  348736
• International Classes
  G06F 017/28
• Field of Search
  364/419.13 364/419.01-419.02 364/419.07-419.08 394/147 382/305 382/306 382/175
• Examiner
  Hayes; Gail O.
• Agent
  Renner, Otto, Boisselle, Sklar
• US Patent References:
  4695975
  4887212
  4999790
  5197005
  5239617
  5265065
  5297027
  5299123
  5303148
  5317508