Indexing, rewriting and efficient querying of relations referencing semistructured data7016910Abstract The invention discloses methods and apparatus that facilitate efficient querying of tables referencing semistructured data such as digraphs and other domains with complex grouping structure. The invention methods enable meaningful indexing of the tables as well as rewriting of queries with respect to the structures. Dynamic schema extraction using proper coloring algorithms is disclosed that structures the semistructured data in such a way that complex set operations and grouping are replaced with traditional relational joins. This enables a relational database system to harness its entire query optimizing capability when querying tables referencing semistructured data. Claims What is claimed is: Description BACKGROUND OF THE INVENTION FIG. 1 shows two digraphs 1001 and 1002 that are used to exemplify the notation described in this disclosure of the invention. The digraphs 1001 and 1002 may also be considered as one disconnected digraph. The nodes in the graphs are labeled A, B, C, D, E, F, G, H, I, J and K and the edges by E(ST) where S is the source node and T the target node. The graph 1001 is directed and acyclic whereas the graph 1002 is cyclic. The present invention deals with both cyclic and acyclic digraphs as well as with non-directed graphs and other equivalent forms and representations of data. Relational/XML Database Representations. FIG. 2 shows example documents containing descriptions of a digraph. In a relational database, graphs may be stored in tables. One of the many ways this can be achieved is to have each row in a given table represent an edge in a graph. This is, for example, the case with relation (table) 2001. The table 2001 contains a description of the (combined) digraph 1001, 1002 shown on FIG. 1. Table 2001 has columns representing the source node, the target note, and the edge. If the naming of edges is irrelevant the table does not need an edge column and may simply be formed as a binary relation over the nodes representing source and target nodes only. Graphs may also be stored in a computer system as text files or other formats such the XML (Extensible Markup Language) document shown as 2002. The files may be entries in tables in a database or imported into relational databases using the various XML or text mappings into relations. The documents may also be stored in native XML databases or XML extensions of relational databases. For techniques relating to native XML databases see the various vendor specific documents such as documents available from NeoCore's webpage: www.neocore.com. For XML extensions of relational databases see the Oracle or IBM references: Oracle9i XML Database Developer's Guide—Oracle XML DB, Release 2 (9.2) March 2002, Part No. A96620-01 or IBM DB2 Universal Database, XML Extender Administration and Programming, Version 8, Part No. CT19TNA. Relational database techniques are discussed in the textbook: Database Management Systems, Second Edition by Raghu Ramakrishnan and Johannes Gehrke, published by McGraw-Hill Higher Education. The SQL standard used in relational database systems is defined by documents: ANSI documents, X3.135-1992, "Database Language SQL" and ANSI/ISO/EIS 9075 available from the American National Standards Institute. A practical vendor specific SQL implementation is described by the Oracle reference: Oracle9i, SQL Reference, Release 2 (9.2), March 2002, Part No. A96540-01 available online from Oracle Corporation, Redwood Shores, Calif., and by the DB2 reference: IBM DB2 Universal Database, SQL Reference Volumes 1 & 2, Version 8, SC09-4844-00 & SC09-4845-00, Parts No. CT17RNA & CT17SNA. The invention also makes references to functions defined inside database systems and both SQL references, above, explain how to create and define such functions. Information and specifications relating to the XML standard is available from the World Wide Web Consortium's (W3C) webpage: www.w3c.org. Search Criteria. In particular, the present invention applies to the following setup. Given a domain, D, i.e., a set of values, and a function F that maps each value to a set of values in D, i.e., for each d in D the output, F(d), is a subset of D. In a relational database system, this function may be represented in many different ways. One way is a table with two columns: One for values d from the domain and another for elements e from the subsets F(d) of D. In other words, the rows in the table contain entries (d,e) where e is in the subset F(d) of D. Such a table defines a binary relation over D. Mathematically, F is a map from D to the powerset of D, i.e., the set of all subsets of D. It is also common in a relational database system to represent such functions by a number or boolean valued function, say f, defined in the database system in such a way that f(e,d)=1 if e is in the set F(d) and f(e,d)=0 otherwise. This is, for example, a common practice in the Oracle database system. The Oracle database system currently, e.g., version 9.2i, allows users to create specialized index methods for "Domain Indexes" to optimize access to relations about the domains. A reference to the technology used by Oracle includes the Oracle handbook: Oracle9i, Data Cartridge Developer's Guide, Release 2 (9.2), March 2002, Part No. A96595-01. Similarly, IBM's Informix Database supports virtual indexes, see the documentation: Virtual-Index Interface, Programmer's Manual, Version 9.3, August 2001, Part No. 000-8345, IBM's Informix Online Documentation, IBM 2001. A somewhat different, but applicable, approach is available as part of DB2's SQL using a "create index extension" statement, see: IBM DB2 Universal Database, SQL Reference Volumes 2, referenced previously, for full documentation. The domain D may also be a composed domain so that each element in D is, for example, a vector containing more than one value. This is a standard indexing technique and the disclosure assumes that an element from the domain, usually called D here, may be structured in different ways. The relation generated by the set valued function F is defined here to be the binary relation over the domain D with entries (d,e) where e is in the set F(d) and d in D. It is referred to as the target relation induced by the set valued function F and denoted by Target(F). It is an objective of the invention to disclose methods and structures that may be used in a relational database system to optimize queries issued on tables containing a column with values from the domain D and wherein the query is partially or entirely specified, i.e., conditioned, using the function F, represented in the database. In order to clarify this with an example, consider the gene ontology digraph defined by the gene ontology consortium, see Gene Ontology: tool for the unification of biology. The Gene Ontology Consortium (2000) Nature Genet. 25: 25-29. Assuming one has imported the publicly available gene ontology digraph into the Oracle database one may proceed and define a function, say Ge(e,d), modeling the previously defined descendants-and-self function in such a way that Ge(e,d)=1 if e is d or a descendant of d, and Ge(e,d)=0 otherwise. An example of a relational SQL query, issued on a table, say goTermFact, with a column "acc", containing entries from the gene ontology digraph and specified using the Ge function has the form: select count(*) from geTermFact where Ge(acc,'GO:0003824')=1 It counts the number of rows in the table geTermFact where the value of the "acc" column is equal to or a descendant of the node 'GO:0003824' in the gene ontology digraph. The difference between the digraph and the relation induced by the function Ge needs to be, and is, emphasized below. FIG. 3 shows examples of 3001, 3002, 3003 several possible set valued node functions for a simple digraph evaluated at a particular node, "B". Each of the functions results in a different induced relation. The first example, 3001, demonstrates a function that maps a node to a set determined by the node itself and its siblings (all the nodes have the same parents). The Sib function of Example 3001 is further represented by the induced intersection graph 4001 of FIG. 4. In that graph 4001, each edge shown connects siblings (nodes of the same parent). Continuing with FIG. 3, the second example shown, 3002, demonstrates a function that maps a node to the descendants-and-self set. The corresponding induced dimension or intersection graph 4002 of FIG. 4 represents the resultant relation of the Ge function of example 3002. In that graph 4002, each edge connects descendant nodes. The third example 3003, in FIG. 3, demonstrates a function that maps a node to its ancestors set. They induced intersection graph 4003 (FIG. 4) illustrates the corresponding relation resulting from function Lt of example 3003. Other such set valued node functions include functions that map a node to its descendants only, its ancestors-and-self set only, its parents, its children or in a weighted graph nodes in a similar weight range, of greater/lesser weight and so on. Additional domain attributes, as in the weight examples, allow one to create countless such maps describing the various physical phenomena. Defining, efficiently, the algorithms required to construct these and other, set valued, node maps, may or may not, be a simple task depending on the definition of the function. The books: The Art of Computer Programming, Volume 3, Sorting and Searching, Second Edition by Donald E. Knuth published by Addison-Wesley (1998) and the book Introduction To Graph Theory, Second Edition by Douglas West, referenced previously, may be used as starting points to the prior art of writing efficient such algorithms. As explained above a domain D, e.g., a finite set of values stored in a database relation, and a set valued function F from D to the powerset of D may be stored in a relational or XML database. In relational databases the function F might be stored or defined by a binary relation over D, i.e., a table with two columns each with values from D. The entries (rows) in the tables are all values of the form (d,e) where e is an element from F(d) and d is in D, as explained above. It has also been explained that the function F may be represented or defined directly as a database function (e.g. using the create function statement), say f, returning numbers or boolean values such that if d and e are values from D then f(e,d)=1 (or TRUE) if e is in F(d) but f(e,d)=0 (or FALSE) otherwise. Yet another alternative is to represent the set valued function by a Boolean condition, e.g., just a string such as "e>d" representing "f(e,d)=1 if e>d, but 0 otherwise". In all of these cases the notation Target(F) or Target(f) may be used. In other words, Target(F) may be regarded as the SQL relation: select d.d as d, e.d as e from D d, D e where f(e.d, d.d)=1 where D is the domain with the nodes in a "d" column and f is the relational database function or a conditional expression, in the latter case "f(e.d, d.d)=1" is replaced with the expression. If the domain D is large then this may be a very inefficient way to define the relation and therefore any additional information about the function may be useful to increase the efficiency of creating the Target(F) table from f. This additional information may be coded into the database as a specialized index extending the indexing capabilities of the the database system such as implemented in the Oracle9i database and previously mentioned. (Alternatively, a more optimal/self-explanatory notation might be: select d.d as d, e.d as e from D d, D e where e.d IN F(d.d)). The above process is demonstrated by algorithm (routine), 5002 in FIG. 5 in connection with other possible efficient algorithms for creating Target(F) explained below. Set Valued Functions Induced by Digraphs. In many cases though the natural way to specify the desired set valued function is to import or define it in the database system using a digraph. For example, given any set valued function F on a domain D, the function F is the target map, Tg, of a digraph obtained by connecting a source node d in D with all the targets in the set F(d). This shows that a target map over a digraph with nodes in D may be used to simulate any such set function. It, and the descendants-and-self function "Ge" as well as the descendants function "Gt" are described in detail below. Equivalently, one can reverse the arrows in the digraph and obtain similar results by describing the "source map", the ancestors-and-self and the ancestors' functions. Another source of set valued functions induced by digraphs comes from using the various path expressions, as will be explained carefully. With reference to FIG. 5 step 5001, given a digraph G represented in a database with nodes from a domain D the induced target relation of the set valued function Tg, denoted by Target(Tg), is obtained as the binary relation (with ordered columns "d" and "e") of all distinct pairs (S,T) where S is a source node and T a target node of an edge in the digraph. This is also the way, in many cases, the original digraph G is realized in the database so no additional work may be required in creating Target(Tg) other than to point to the original digraph. Creating Target(F) from a function or a logical conditional expression is explained previously. The algorithms 5001 and 5002 on FIG. 5 summarize the steps required to create Target(Tg) and Target(F) from a function or conditional expression. The induced Target(Gt) relation for the digraph induced by the descendants set function (gt above) may be defined by the following algorithm as illustrated at 6001 in FIG. 6. Target(Gt): Start with an empty Target(Gt) relation with ordered attribute headings "d" and "e". Initialize Target(Gt) by adding all the ordered edge endpoints to Target(Gt), i.e., add all pairs (S,T) where S is the source node of an edge and T is the target node, excluding repetitions of such pairs. The process continues by iterating the following step: For each of the entries (S,T) added to Target(Gt) in the previous step (initialization being the first step) add all, not already existing, entries to Target(Gt) of the form (S,X) where X is a target node of an edge in the digraph with source node equal to T. The process should be stopped when the foregoing step results in no more additions to the Target(Gt) relation as illustrated at the end loop of 6001 in FIG. 6. The above algorithm can be efficiently executed in a relational database system supporting simple programming and indexing of (e.g. B-tree) tables. This is the case both with IBM's DB2 and the Oracle database. Similarly it may be efficiently executed in an XML extension or in a native XML database supporting indexing and minimal programming. If one adds a loop to each node in the digraph then each node becomes a descendant of itself and Gt morphs into Ge. Nevertheless, the search graph for the descendants-and-self function, Ge, over the digraph may be defined by a similar independent algorithm as follows as illustrated at 6002 in FIG. 6. Target(Ge): Start with an empty Target(Ge) relation with ordered attribute headings "d" and "e" as before. Initialize Target(Ge) by adding all entries of the form (N,N) to Target(Ge) where N is a node in the graph. The process now continues in the same way as before by iterating the following step: For each of the entries (S,T) added to Target(Ge) in the previous step (initialization being the first step) add all, not already existing, entries to Target(Ge) of the form (S,X) where X is a target node of an edge in the digraph with source node equal to T. Again, this should continue until the foregoing step results in no more additions to the Target(Ge) relation as illustrated at the end loop of 6002 in FIG. 6. The Target(Ge) relation may additionally be obtained from Target(Gt) by adding all entries of the form (N,N) with N a node in the digraph, not already included in the Target(Gt) relation, i.e., Target(Ge)=Target(Gt) "union" the diagonal line in the cross product of D with itself. The above two basic algorithms for creating Target(Gt) and Target(Ge) from a digraph are illustrated on FIG. 6 by 6001 and 6002, respectively. Path Expressions and Filtering. A rich source of set valued functions is obtained from path expressions. Path expressions are supported in many database systems and can thus be efficiently evaluated using techniques already available in the systems. Path expressions are discussed in the previously mentioned text: Data on the Web, From Relations to Semistructured Data and XML by Serge Abiteboul, Peter Buneman and Dan Suciu. A standard called the XML Path Language (XPath) has been developed for path expressions in XML, within the World Wide Web Consortium. Common, search related, path expressions provide specifications which point to nodes in digraphs. The syntax used for path expressions varies from system to system. As an example, the path expression "d:._*" may be used to specify the descendants-and-self map Ge(d) described previously, and the path expression "d:._._*" may be used to define the descendants set Gt(d) for a given node d. Explicitly, in this example, the expression "d:._._*" results in all nodes that can be reached starting from the node d and following at least one edge in the direction of the digraph, similarly "d:._*" specifies all nodes that can be reached starting from d and following zero or more edges forward in the digraph. The set valued function, F(d), associated with a path expression specified as a function on the domain D, may be defined as explained below and accordingly realized in a database system: As a further example, the "geneology" expression, exp(e,d)="e:.mother._*.d:", may be used to specify the set valued function F(d)={e|EXISTS(e:.mother._*.d:)"}. The set F(d) specifies the "mother", "grandmothers" and so on for the node "d". A database system may provide support for path expressions, in which case the associated set valued function will be efficiently implemented using the supported features and indexing. Intersection Graphs Induced by Set Valued Function. For a set valued function F over a domain D, the Target(F) relation induced by F may be efficiently defined in a database system according to the invention, by the above description. The intersection graph of the set valued function F, denoted by Int(F), is now defined here as follows: In graph theoretical terms, the family of sets F(d), for d in D, forms an intersection representation of the graph Int(F) and thus Int(F) is called the intersection graph of the family of sets, but here calling Int(F) the intersection graph of F will do. The Int(F) graph will also be referred to as the intersection graph induced by Target(F) (and D). As mentioned above, FIG. 4 shows the intersection graphs for the 3 set valued functions (Sib, Ge, Lt) defined from the examples 3001, 3002, 3003 shown on FIG. 3. The graph Int(Sib) is labeled 4001, the graph Int(Ge) is labeled 4002 and the graph Int(Lt) is labeled 4003. The existence of the edges shown is quickly verified from the illustrations of corresponding functions (Sib, Ge, Lt) in FIG. 3. Proper Coloring of the Int(F) Graph. Let F be a set valued function on a domain D, defined directly in the database or through the use of a digraph represented in the database as described above. The proper coloring of the graph Int(F) may be efficiently achieved in a database system. The theory of graph coloring is discussed in the book: Introduction To Graph Theory, Second Edition by Douglas West referenced earlier. Other references include the books: Graph Coloring Problems by Tommy R. Jensen and Bjarne Toft and published by John Wiley & Sons, Inc. (1995) and the text Graph Colouring and the Probabilistic Method by Michael Molloy and Bruce Reed published by Springer Verlag (2002). A discussion about the chromatic number of the graph, Int(Ge), for specific classes of digraphs is contained in the preprint: On vertex coloring simple digraphs by Geir Agnarsson and Agust Egilsson [2002]. In one embodiment, a greedy proper coloring algorithm 7001 is used to color the graph Int(F) by looping over the nodes as follows and illustrated in FIG. 7: Select the nodes from D in some order. For each selected node, d, assign to it the smallest positive integer, k (the color of d), such that none of its neighbors has already been assigned the same color k. In a more machine/SQL friendly manner the algorithm 7001 may be implemented as follows for the Int(F) graph:
The greedy proper coloring algorithm is demonstrated as routine 7001 on FIG. 7 for purposes of illustration and not limitation. There are many ways to write proper coloring algorithms for the Int(F) graph as evident by the above graph coloring references. The best algorithms will efficiently produce coloring using only close to the minimal numbers of colors for specific types of graphs, i.e., the chromatic numbers of the graphs. For the descendants-and-self function and graphs such as the Gene Ontology digraph (approximately 11,000 nodes), referenced above, the greedy algorithm 7001, above, on the other hand suffices (currently) and efficiently results in a coloring using the minimum number of colors (36 as of Fall 2002—using increasing ordering of the nodes). The Clique(F) Relation. It has been disclosed in the above sections how to efficiently obtain in a database system the Target(F) relation and the Color(F) relation induced by a set valued function F over a domain D. The structures revealed in the Int(F) graph and its proper coloring, Color(F), may be used to create and optimize access plans to relations referencing the domain D. One way to take advantage of the Int(F) graph and the Color(F) relation is to extract a schema, denoted here by Clique(F), that may be used to optimize querying, as defined below: The Clique(F) relation: Start with an empty relation Clique(F) with columns to represent the nodes in the Int(F) graph: One reference column (denoted here by "node") and additional columns representing each of the colors used in the coloring relation Color(F)—(denoted here by "C1", "C2", . . . , "Cn" where n is the number of colors used). Each of the nodes in the domain D is assigned a single row in the relation Clique(F) in such a way that the node itself, call it e, is mapped to the "node" column and each of the nodes d satisfying the condition: (d,e) is in Target(F) is mapped to the column representing the color of d, i.e., the color k where (d,k) is in Color(F). The remaining slots in the row may be left empty (i.e., contain the "NULL" attribute in most database systems). Consequently, the Clique(F) relation contains rows (e,D(e,1), . . . , D(e,n)) where e is from the domain D and n is the number of colors, the slot D(e,k) is empty or references a node d if (d,e) is in the relation Target(F), induced by F, and k is the color of d, i.e., (d,k) is in Color(F). A formal definition is therefore given by: D(e,k)=d if (d,e) is in Target(F) and (d,k) is in Color(F), D(e,k) is empty if no such d exists. For any fixed e, the set of nodes d satisfying: (d,e) is in Target(F), form, by definition of the Int(F) graph, a clique in the graph and therefore are all assigned different colors by any proper coloring algorithm. The algorithm for creating the Clique(F) relation is illustrated on FIG. 8 by flowchart 8001 as well as detailed above. It may be implemented efficiently in a relational or XML database system supporting minimal programming. FIG. 9 shows a digraph identified by 9001. The, set valued, map descendants-and-self, derived from this digraph, is used to exemplify the above concepts. Firstly, the relation Target(Ge) induced by the descendants-and-self map is shown as 10001 on FIG. 10. Then the intersection graph Int(Ge) induced by the map is shown as 10002. The Int(Ge) graph need not be constructed in the database directly but its definition is used by the proper coloring algorithm. The result of a proper coloring algorithm (e.g. 7001) applied to the graph, 10002, is shown on FIG. 11 as relation 11001 and also identified by Color(Ge). Finally, extraction algorithm 8001 defined above is exemplified by relation 11002 and also identified by "Clique(Ge)" on FIG. 11 showing the result of the algorithm when applied to the descendants-and-self map for the particular case of the digraph 9001 shown on FIG. 9. Similarly, the results of the algorithm applied to the other set valued maps, identified previously by, Gt, Le, Lt applied to digraph 9001, are shown as "Clique(Gt)" 12001, "Clique(Le)" 12002 and "Clique(Lt)" 13001 respectively, on FIGS. 12 and 13. The General Idea. As explained earlier the schemas extracted, i.e., Clique(F), are used to add structure to large relations so that optimal access plans may be generated and executed in a database system. In particular the following applies: Given a set valued function F on a domain D, as above. Denote by "FactTable" a (possibly very large) relation in the database system that references the domain D in one of its columns, e.g., "node", containing entries from the domain D. A query accessing or analyzing information from the table using a set expression, to condition the query, equivalent to: where d is a node from D, is now equivalent to the following relational expression: "FactTable.node=Clique(F).node and Clique(F).Ck=d" (3) where Ck is the column representing the color (k) of d in Clique(F). When creating and executing access plans, form (3) reveals additional relational structure that may be used to evaluate the query efficiently. It enables the use of star-transformations, i.e., specific optimization methods for this equation (3) and similar settings and the use of materialized views. Form (3) also enables the use of many additional indexing techniques, including the use of bitmap and bitmap join indexing which may dramatically increase the performance of the query. See for example the documents: Oracle9i, Data Warehousing Guide, Release 2 (9.2), March 2002, Part No. A96520-01 or the Oracle9i, SQL Reference mentioned earlier for a discussion about the various access methods. The expression "Clique(F).Ck=d" used in equation (3) may be replaced with a more complicated statement not requiring any information about the color (k) of d in Clique(F). It is, for example, equivalent to "(Clique(F).C1=d OR Clique(F).C2=d OR . . . OR Clique(F).Cn=d)" where the expression is repeated for all colors from 1 to n (the number of colors used). It will in some cases, though, require more processing effort not to include information about the coloring in this way. The example on FIG. 14 shows a "FactTable" 14001, the table Clique(Ge) 14002 (also denoted by 11002 on FIG. 11) and an equijoin operation 14003 required to connect to the Clique(Ge) structure. In order for the database to be able to take advantage of the relationship between the tables, it may be necessary to identify the entries in the "node" column in the Clique(Ge) relation as unique. It may also be necessary to hint or otherwise inform the database system about the structure of the Clique(Ge) table. Query Rewrite. A system may take advantage of the schema extracted, Clique(F), and the proper coloring of the Int(F) graph by simply translating queries that reference the function or expression, F (or f, etc), into equivalent queries using Clique(F) and the coloring. As explained above the statement "f(FactTable.node, d)=1" is translated into "FactTable.node=Clique(F).node and Clique(F).Ck=d" where k is the color of the node d. As a further explanation, a previously mentioned query, (A) select count(*) from geTermFact where Ge(acc,'GO:0003824')=1 may be transformed into the query (B) select count(*) from geTermFact fact, Clique(Ge) clique where fact.node=clique.node and clique.C8='GO:0003824' Assuming that the node GO:0003824 has been assigned color 8 by the proper coloring algorithm used to create Clique(Ge). It is to be understood, as always in similar cases, that a valid database name has to be assigned to the relation identified by Clique(Ge)—in the above example Ge is the descendants-and-self map for the gene ontology digraph. In evaluating the query (B), the database system may select from several possible access plans. In some cases a bitmap join index may have been defined in the database system on the relationship between the goTermFact and the Clique(Ge) table for queries referencing column C8 and joining the node columns. This particular query may in that case be evaluated without accessing either of the tables but instead a bitmap array corresponding to the node GO:0003824 may be used instead, resulting in an efficient evaluation of the query even for the largest of tables. Another convenient way to hide all the details and transformations from the users and systems accessing the information in the database is to use extendable or native indexing in the database taking advantage of the structures. This approach is explained below. Extendable and Native Indexing. Querying relations based on the entries in columns when evaluated by a function or based on position in a digraph may be effectively achieved using the structures disclosed. The process can be automated by taking advantage of extendable or native indexes inside database systems. There are several options when constructing the index methods. Firstly, the index constructed may return lists of: Secondly, the input for the index-create method may require a digraph, a function or a conditional expression to construct the index over a table column. Some of the options facing the index designer include: In the first two cases, the techniques required to create the additional structures: the Clique and the Color relation, have been disclosed. The third format requires the domain D to be defined as the (distinct) values coming from the table column(s) and requires the Clique and Color relation to be maintained dynamically. This is discussed in the section on variable domains below. The use of additional database structures such as bitmap join indexes has also been disclosed. The index-create method may therefore set up, the schemas extracted from the semistructured data, the Clique and Color relation as well as to establish additional indexing both on the tables individually and by using the join condition between the table column(s) and the Clique table. This may include bitmap join indexes. One of the current implementations of the system in an Oracle database, for example, creates 36 bitmap join indexes (since there are 36 colors required for proper coloring of Int(Ge) in this case) when indexing a column referencing the gene ontology digraph. Queries using the function take full advantage of these bitmap join indexes through the use of extendable indexing in Oracle. When queries are issued that are conditioned by a function/operator and a column that has been indexed by the extendable indexing or by native indexing technologies, the system may rely on the indexing to provide the resulting rowids or bitmaps. It is then the responsibility of the indexing technology to use the proper coloring and the Clique tables to construct a query taking advantage of the additional structures extracted and additional indexing set in place, and maintained by the indexing methodology. The methodology created to maintain indexes and examples are disclosed in the Oracle document: Oracle9i, Data Cartridge Developer's Guide. Variable Domains. The domain, D, used to denote the input for the set valued function is in many cases not known beforehand or is deemed too large. It may for example just be the set of all numbers available in a database system. In this case, the domain may be derived dynamically and updated from the table being indexed directly so that it contains only a small subset of all possible values. The domain D is in this case referred to as being variable. Since the set valued function F is now defined on a domain which is allowed to change, the definition of the function may be required to be deterministic in nature, i.e., the value f(e,d) does not depend on the other elements in the domain, only on the input values "e" and "d". The induced relation, Target(F) and the structures Clique(F) and Color(F) may be maintained dynamically as the domain varies. The two operations that need to be implemented are: In the preferred embodiment, the incremental algorithms required in each operation are as follows with reference to FIG. 15, many variations are possible though: Adding a new element to D: The algorithms required to modify Target(F), Color(F) and Clique(F) to accommodate a new element, say Q, are explained below. It is assumed that the relations D, Target(F), Color(F) and Clique(F) are all synchronized (in a consistent state with respect to the domain D and the set valued function F). After the new node, Q, has been added to the domain and all the relations have been updated the corresponding synchronized relations are denoted by D+, Target(F)+, Color(F)+ and Clique(F)+. Additionally, the intersection graph induced by D+ and Target(F)+ is referred to as Int(F)+, as before it need not be explicitly realized in the database. As always, there are many possible equivalent variations of the processes defined:
This is illustrated in step 15002 in FIG. 15. The recoloring may be achieved as follows: Start by determining a relation mapping the old color of some of the nodes to new colors. This may involve the following steps:
It references all the nodes (except possibly Q itself) that are required to construct the row in Clique(F)+ corresponding to Q. Therefore all these nodes need to be assigned different colors, if that is not the case already.
The modified relation Clique(F) is denoted by Clique(F)+ and it is now synchronized with the other relations D+, Target(F)+ and Color(F)+ as required. Of course, the relations need not be represented in a database system. One may quite as well build and maintain the objects using almost any computer language and system. The algorithms outlined above in items 1, 2, 3 and 4 are summarized on FIG. 15 as 15001, 15002, 15003 and 15004, respectively. Removing an existing element from D: The algorithms required to modify Target(F), Color(F) and Clique(F), when an element is removed from D are explained below with reference to FIG. 16. Again, it is assumed that the relations D, Target(F), Color(F) and Clique(F) are all synchronized before the process starts. After the node has been removed the corresponding synchronized relations are denoted by D-, Target(F)-, Color(F)- and Clique(F)-. The element to be removed from the domain will be denoted by the letter P. There are many possible equivalent variations of the processes outlined. These two steps are efficiently implemented in SQL using simple "DELETE" statements. They may also be deferred without affecting the logic of the system.
As indicated it is not necessary to perform the above steps 1 to 4 every time a node is removed. A bulk removal is acceptable in most cases. Periodically, a recoloring or partial recoloring and cleanup, may be applied to make the Clique(F) table more compact after one or several nodes have been removed. The processes described in steps 1 to 4 above are summarized on FIG. 16 as process 16001. The above disclosed algorithms are used for dynamically maintaining the extracted schema structures as explained. They may therefore be used to dynamically maintain indexes that efficiently facilitate complex grouping of values in a column. Such an index may be understood to be a set-valued-function/multivariable-expression index or simply a grouping index. Each value x on the domain defines a group of values, i.e., F(x). This is further demonstrated in the examples below. Variations. There are many equivalent ways to implement the methods disclosed as is apparent to the person skilled in the art. In some cases, system limitations require alternative implementations. One such limitation in relational database systems is the maximum number of columns that may be used in a table, e.g., approximately 1000 in Oracle 9i. In cases when the number of colors needed to properly color the induced Int(F) graph exceeds this number, the Clique(F) table may broken into several tables each representing only a subset of the colors, i.e., using vertical fragmentation. It is also possible to keep some of the performance enhancements associated with using the extracted Clique(F) schema without using any proper coloring at all, thereby obtaining a more compact structure. An example of such a design is shown as 13002 in FIG. 13 and labeled "Ge—Multicolor". Each row in the table contains the same nodes as shown in table 11002 on FIG. 11, but columns in 13002 do not represent colors of nodes and the same node is assigned to more than one column in some cases. This structure, i.e., 13002, can also be joined with a table referencing graph 9001 on FIG. 9, and queries utilizing the Ge operator may be efficiently translated to queries about the joined tables. The multicolor setup, just described, also applies in general to the algorithms disclosed in the application. The Target(F) relation, or equivalent structures, can additionally be used directly to build set-valued-functional indexes on a relation referencing semistructured data as follows. For each node d in the domain D, the extracted Target(F) table is used to build, on demand or permanently, bitmap arrays pointing to all rows in the relation containing nodes from F(d). This may be achieved by using the database system to build bitmap indexes on the referencing column(s) in the relation directly and then use the logical OR operator to generate bitmap arrays that represent rows with elements from the set F(d). In other words, by applying the logical OR operator to all the bitmap arrays pointing to rows in the relation containing individual nodes from F(d), e.g., using Target(F) to obtain such a list of nodes. The resulting composed bitmap arrays may be maintained and used by the database system as part of a set-valued-functional index definition. Additional Usage Examples.
The conditional statement used "ln (y)*x>cos (x)+y" is a Boolean statement that may be used to populate Target(F) as described earlier and therefore generate and maintain Clique(F). The first part "y is in F(x) iff:" is used to determine what are the variables used in the description. No digraph is required and the index may be maintained dynamically using the algorithms disclosed earlier. Using the index is simple, e.g., using current Oracle 9.2i indexing methodology, the index-type is associated with a function f(y,x) so that a query such as:
The usefulness of the index is particularly clear when the ratio between the number of rows in the Observations table and the distinct values (domain D) on the x column is high. Instead of using the complicated formula above, the indexing joins the Clique(F) table with the Observations table (f above) so that the database system can take advantage of the equivalence between:
Similarly to the previous example the expression "sqrt((x-a)*(x-a)+(y-b)*(y-b))<10" may be used to build Target(F) and consequently therefore also Clique(F). In many cases a filtering hint submitted will increase the efficiency of inserts into the table, i.e., the maintenance of Target(F), in this case the pre-filtering may be submitted by replacing the formula "sqrt((x-a)*(x-a)+(y-b)*(y-b))<10" with the equivalent formula: "a<x+10 and a>x-10 and b<y+10 and b>y-10 and sqrt((x-a)*(x-a)+(y-b)*(y-b))<10". This is accomplished using the index create expression:
Depending on how clever the database system is, the filtering hints may be expanded further, e.g., "a<x+10" may be replace with "a<x+10" and "x>a-10" and so on. The index may now be used to evaluate efficiently queries, relating to accidents and neighborhoods, through an index-type binding with some operator f, such as:
The index create statement, when executed, colors the induced intersection graph and builds the extracted Clique(Ge) relation shown as 17002 on FIG. 17. The Target(Ge) relation used in the process is efficiently built using the digraph as disclosed above and the coloring of Int(Ge) proceeds in a greedy fashion as previously disclosed. The Int(Ge) graph requires 5 colors represented by columns C1, C2, C3, C4 and C5 in table Clique(Ge). The index create statement also adds bitmap join indexes on the color columns based on the equijoin between the node columns of Clique(F) and the Measurements table. This may be done using the bitmap join index create statement of the relational database system. Queries using the Ge operator are then made to take advantage of the bitmap join indexes by the indexing schema created. The rowids of the Measurements table satisfying a query such as:
The index may also be instructed to select other access plans, such as other star transformations not involving the use of bitmap join indexes. Both a regular bitmap index and a bTree index on the node column in the Measurements table can be utilized. The Clique(Ge) relation is small and the mapping from the color columns to the node column in Clique(Ge) is most efficiently handled using in-memory operations, and in-memory derived structures, when an access plan requires such a mapping. Correctly set cost parameters will allow the database to select the most efficient access plan automatically based on available additional indexes.
It will be clear to a person skilled in the art that the methods disclosed in the example and in the above may be used to create a bitmap indexing system for path expressions. CONCLUSION The invention may be implemented as any suitable combination of hardware and software. FIG. 18 is a diagram of a computer architecture for implementing embodiments of the present invention. An application 202, 204 running on client computers 206, 208 is connected to a communications network 210. The application 202, 204 can be a Web Browser (e.g., Netscape Navigator or Microsoft Internet Explorer) or a proprietary client. The communications network 210 can be a proprietary network (e.g., Local area network, wide area network, etc.), a public network (e.g., Internet) or some combination of both. The communications network 210 connects client computers 206, 208 to server computer 212. Server application programs 216, run on the server 212 and provide access to data stored in connected database 220. Application programs 216 may be server software or other Internet server software or the like. A database 220 contains records 226, forms 230, and UI components 232. Database records 226 store data in fields. Forms 230 define the layout, either storage or presentation, of data stored in the database 220. UI components 232 are stored with the database 220 and provide various controls for interacting with the database records 226. The present invention method and/or apparatus may be implemented at 228 as part of the database system 220 or at the application level 216, for example. Although client applications (202, 204) shown in FIG. 18 are connected through a network 210, they also may be locally connected directly to database 220, 222, so that network 210 is not required. While particular embodiments have been described, various other modifications will be apparent to those skilled in the art. While this invention has been particularly shown and described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.
|
Same subclass Same class Consider this |
||||||||||