System and method for intelligent caching and refresh of dynamically generated and static web content

Abstract

A system for updating Web pages stored in cache based on modifications to data stored in a database is disclosed. The system for updating stored Web pages may be part of a larger system having a database management system for storing data used to generate Web pages. The database management system is capable of identifying modified data stored in the database. The system for updating stored Web pages is comprised of one or more servers programmed for maintaining associations between the stored Web pages and the stored data, and receiving the identity of modified data from the memory management system. In addition, the servers are capable of determining, from the identified modified data and the maintained associations, which stored Web pages are associated with the identified modified data. Furthermore, the servers are capable of communicating an update command to the cache that contains the stored Web pages associated with the identified modified data, for the purpose of updating the stored Web pages.

Claims

What is claimed is:

1. In a system having at least one first memory for storing Web pages and a memory management system including at least one second memory for storing data used to generate Web pages, the memory management system capable of identifying modified data stored in the at least one second memory and maintaining at least one database log sequentially containing all operations to the at least one second memory, a system for updating stored Web pages based on modifications to data stored in the at least one second memory, comprising:

at least one server programmed for

maintaining associations between each stored Web page and the stored data used to generate that particular Web page,

receiving the identity of modified data from the memory management system,

determining, from the identified modified data and the maintained associations, which stored Web pages are associated with the identified modified data, and

communicating an update command to the at least one first memory that contains the stored Web pages associated with the identified modified data for updating the stored Web pages.

2. A system for updating stored Web pages as recited in claim 1, the at least one server further programmed for:

receiving a Web content request containing a URL;

issuing at least one query or operation to the at least one second memory in response to the Web content request;

maintaining associations between the URL of the Web content request and the issued queries or operations;

maintaining associations between the issued queries or operations and the stored data accessed by the queries or operations;

deriving associations between the stored Web pages and the stored data from the maintained associations between the URL of the Web content request and the issued queries or operations, and the maintained associations between the issued queries or operations and the stored data accessed by the queries or operations; and

identifying, through prior processing steps to store the stored Web pages in the at least one first memory, the location of the Web pages associated with the identified modified data.

3. A system for updating the stored Web pages as recited in claim 2, wherein the update command includes a message to invalidate or delete the stored Web pages.

4. A system for updating the stored Web pages as recited in claim 2, wherein the update command includes a message to refresh the stored Web pages.

5. A system for updating the stored Web pages as recited in claim 4, wherein if the stored Web page is in a Web server, the update command includes a message to generate a new Web page and store the new Web page in the Web server.

6. A system for updating stored Web pages as recited in claim 2, the at least one server further programmed for:

maintaining a Web server log containing a request and delivery timestamp for the received URL;

maintaining a query and operation log containing an issuance timestamp for the issued query or operation;

deriving associations between the stored Web pages and the stored data from the Web server log and the query and operation log; and

determining, from the identity of the modified data and the derived MO associations between the stored Web pages and the stored data, which stored Web pages are associated with the identified modified data.

7. A system for updating stored Web pages as recited in claim 6, the at least one server including:

a Web server programmed for receiving the Web content request and maintaining the Web server log;

an application server programmed for receiving a forwarded Web content request from the Web server and issuing the at least one query or operation to the at least one second memory in response to the forwarded Web content request;

a content change monitor programmed for maintaining the associations between the issued queries or operations and the stored data accessed by the queries or operations, and determining which queries or operations are associated with the identified modified data; and

a content delivery services server programmed for determining the associations between the URLs of the Web content requests and the issued queries or operations from the query and operation logs and the Web server log, receiving an identity of the queries or operations associated with the identified modified data from the content change monitor, determining which stored Web pages are associated with the queries or operations associated with the identified modified data, and communicating the update command to the at least one first memory that contains the stored Web pages associated with the identified modified data.

8. A system for updating the stored Web pages as recited in claim 7, the content change monitor programmed for:

receiving queries sent to the at least one second memory;

defining a view table for each query containing pointers to all data in the at least one second memory that is associated with that particular query;

receiving the identified modified data from the memory management system; and

determining from the view table and the identified modified data any queries associated with the identified modified data.

9. A system for updating the stored Web pages as recited in claim 7, the content change monitor programmed for:

receiving operations sent to the at least one second memory;

defining a daemon for each operation, the daemon establishing criteria for identifying stored data in the at least one second memory that is associated with that particular operation; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the daemon.

10. A system for updating the stored Web pages as recited in claim 7, the content change monitor programmed for:

receiving queries sent to the at least one second memory;

creating a trigger definition for each query, the trigger definition establishing criteria for identifying stored data in the at least one second memory associated with that particular query; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the trigger definition.

11. A system for updating the stored Web pages as recited in claim 7, the content change monitor containing at least one third memory and further programmed for:

replicating data from the at least one second memory to the at least one third memory to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

12. A system for updating the stored Web pages as recited in claim 8, the content change monitor containing at least one third memory and further programmed for:

performing all operations in the database log on the at least one third memory, in sequential order, to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

13. A system for updating the stored Web pages as recited in claim 6, further including a sniffer for listening to network traffic by reading request/delivery logs, collecting URL information and query or operation instances by reading a query instance delivery log, and generating the associations between the URL of the Web content request and the issued queries or operations.

14. A system for updating the stored Web pages as recited in claim 6, further including an invalidator/refresher for notifying appropriate caches or Web servers of the invalidity of a particular stored Web page, the invalidator/refresher including:

a registration module for creating invalidation policies and collecting relevant query or operation statistics;

an invalidation module for reading the database update logs and the associations between the URL of the Web content request and the issued queries or operations, scheduling subqueries, and notifying appropriate caches or Web servers of the invalidity of a particular stored Web page; and

an information management module for creating auxiliary data structures for the invalidation module.

15. A system for updating the stored Web pages as recited in claim 6, the at least one server further programmed for:

receiving a query sent to the at least one second memory, the query including one or more query conditions linked by operators, the linked query conditions identifying data in the at least one second memory associated with that query; and

determining if the query is associated with the identified modified data by determining if the identified modified data satisfies the linked query conditions.

16. A system for updating the stored Web pages as recited in claim 15, wherein if the linked query conditions require access to more than one table in the at least one second memory, the at least one server is further programmed such that for each table to which access is required, the at least one server determines if the identified modified data satisfies those portions of the linked query condition specific to that particular table;

wherein after it is determined whether the identified modified data satisfies those portions of the linked query condition specific to a particular table, those portions of the linked query condition specific to a particular table are replaced by a determined value prior to evaluating those portions of the linked query condition specific to another particular table; and

wherein, if at any time during the determination of whether the identified modified data satisfies those portions of the linked query condition specific to a particular table, it is determined that the linked query conditions are satisfied, then the identified modified data satisfies the linked query conditions.

17. A system for updating the stored Web pages as recited in claim 16, wherein after those portions of the linked query condition specific to a particular table are evaluated and a value for those portions is determined, in order to evaluate those portions of the linked query condition specific to another particular table, the at least one server is further programmed for:

issuing a subquery to the other particular table containing the linked query conditions, with portions of the linked query condition replaced by any previously determined values.

18. A system for updating the stored Web pages as recited in claim 17, wherein after a subquery issued to a particular table returns determined values for that subquery, the determined values are stored in an index structure;

wherein if subsequent queries are received by the at least one server that would require reissuing the same subquery, the index structure may be accessed by the at least one server to avoid having to issue the same subquery.

19. A system for updating stored Web pages as recited in claim 2, the at least one server further programmed for:

maintaining a URL/relevant operation mapping log from the URLs contained in the received Web content requests and the queries or operations issued to the least one second memory in response to the Web content requests; and

determining, from the identity of the modified data, the maintained associations between the issued queries or operations and the stored data, and the URL/relevant operation mapping log, which stored Web pages are associated with the identified modified data.

20. A system for updating stored Web pages as recited in claim 19, the at least one server including:

an application server programmed for maintaining the URL/relevant operation mapping log, receiving a forwarded Web content request and issuing the at least one query or operation to the at least one second memory in response to the forwarded Web content request;

a content change monitor programmed for maintaining the associations between the issued queries or operations and the stored data accessed by the queries or operations, and determining which queries or operations are associated with the identified modified data; and

a content delivery services server programmed for receiving an identity of the queries or operations associated with the identified modified data from the content change monitor, receiving the URL/relevant operation mapping log, determining which stored Web pages are associated with the queries or operations associated with the identified modified data, and communicating the update command to the at least one first memory that contains the stored Web pages associated with the identified modified data.

21. A system for updating the stored Web pages as recited in claim 20, the content change monitor programmed for:

receiving queries sent to the at least one second memory;

defining a view table for each query containing pointers to all data in the at least one second memory that is associated with that particular query;

receiving the identified modified data from the memory management system; and

determining from the view table and the identified modified data any queries associated with the identified modified data.

22. A system for updating the stored Web pages as recited in claim 20, the content change monitor programmed for:

receiving operations sent to the at least one second memory;

defining a daemon for each operation, the daemon establishing criteria for identifying stored data in the at least one second memory that is associated with that particular operation; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the daemon.

23. A system for updating the stored Web pages as recited in claim 20, the content change monitor programmed for:

receiving queries sent to the at least one second memory;

creating a trigger definition for each query, the trigger definition establishing criteria for identifying stored data in the at least one second memory associated with that particular query;

determining queries associated with the identified modified data by determining if the identified modified data satisfies the trigger definition.

24. A system for updating the stored Web pages as recited in claim 20, the content change monitor containing at least one third memory and further programmed for:

replicating data from the at least one second memory to the at least one third memory to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

25. A system for updating the stored Web pages as recited in claim 21, the content change monitor containing at least one third memory and further programmed for:

performing all operations in the database log on the at least one third memory, in sequential order, to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

26. A system for updating the stored Web pages as recited in claim 19, further including a sniffer for listening to network traffic by reading request/delivery logs, collecting URL information and query or operation instances by reading a query instance delivery log, and generating the associations between the URL of the Web content request and the issued queries or operations.

27. A system for updating the stored Web pages as recited in claim 19, further including an invalidator/refresher for notifying appropriate caches or Web servers of the invalidity of a particular stored Web page, the invalidator/refresher including:

a registration module for creating invalidation policies and collecting relevant query or operation statistics;

an invalidation module for reading the database update logs and the associations between the URL of the Web content request and the issued queries or operations, scheduling subqueries, and notifying appropriate caches or Web servers of the invalidity of a particular stored Web page; and

an information management module for creating auxiliary data structures for the invalidation module.

28. A system for updating the stored Web pages as recited in claim 19, the at least one server further programmed for:

receiving a query sent to the at least one second memory, the query including one or more query conditions linked by operators, the linked query conditions identifying data in the at least one second memory associated with that query; and

determining if the query is associated with the identified modified data by determining if the identified modified data satisfies the linked query conditions.

29. A system for updating the stored Web pages as recited in claim 28, wherein if the linked query conditions require access to more than one table in the at least one second memory, the at least one server is further programmed such that for each table to which access is required, the at least one server determines if the identified modified data satisfies those portions of the linked query condition specific to that particular table;

wherein after it is determined whether the identified modified data satisfies those portions of the linked query condition specific to a particular table, those portions of the linked query condition specific to a particular table are replaced by a determined value prior to evaluating those portions of the linked query condition specific to another particular table; and

wherein, if at any time during the determination of whether the identified modified data satisfies those portions of the linked query condition specific to a particular table, it is determined that the linked query conditions are satisfied, then the identified modified data satisfies the linked query conditions.

30. A system for updating the stored Web pages as recited in claim 29, wherein after those portions of the linked query condition specific to a particular table are evaluated and a value for those portions is determined, in order to evaluate those portions of the linked query condition specific to another particular table, the at least one server is further programmed for:

issuing a subquery to the other particular table containing the linked query conditions, with portions of the linked query condition replaced by any previously determined values.

31. A system for updating the stored Web pages as recited in claim 30, wherein after a subquery issued to a particular table returns determined values for that subquery, the determined values are stored in an index structure;

wherein if subsequent queries are received by the at least one server that would require reissuing the same subquery, the index structure may be accessed by the at least one server to avoid having to issue the same subquery.

32. A system for updating stored Web pages as recited in claim 19, the at least one server including:

an application server programmed for receiving a forwarded Web content request and issuing a content delivery services provider API call;

a content delivery services provider API handler programmed for receiving the content delivery services provider API call, maintaining the URL/relevant operation mapping log, and issuing the at least one query or operation to the at least one second memory in response to the content delivery services provider API call;

a content change monitor programmed for maintaining the associations between the issued queries or operations and the stored data accessed by the queries or operations, and determining which queries or operations are associated with the identified modified data; and

a content delivery services server programmed for receiving an identity of the queries or operations associated with the identified modified data from the content change monitor, receiving the URL/relevant operation mapping log, determining which stored Web pages are associated with the queries or operations associated with the identified modified data, and communicating the update command to the at least one first memory that contains the stored Web pages associated with the identified modified data.

33. A system for updating the stored Web pages as recited in claim 32, the content change monitor programmed for:

receiving queries sent to the at least one second memory;

defining a view table for each query containing pointers to all data in the at least one second memory that is associated with that particular query;

receiving the identified modified data from the memory management system; and

determining from the view table and the identified modified data any queries associated with the identified modified data.

34. A system for updating the stored Web pages as recited in claim 32, the content change monitor programmed for:

receiving operations sent to the at least one second memory;

defining a daemon for each operation, the daemon establishing criteria for identifying stored data in the at least one second memory that is associated with that particular operation; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the daemon.

35. A system for updating the stored Web pages as recited in claim 32, the content change monitor programmed for:

receiving queries sent to the at least one second memory;

creating a trigger definition for each query, the trigger definition establishing criteria for identifying stored data in the at least one second memory associated with that particular query; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the trigger definition.

36. A system for updating the stored Web pages as recited in claim 32, the content change monitor containing at least one third memory and further programmed for:

replicating data from the at least one second memory to the at least one third memory to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

37. A system for updating the stored Web pages as recited in claim 33, the content change monitor containing at least one third memory and further programmed for:

performing all operations in the database log on the at least one third memory, in sequential order, to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

38. A system for updating stored Web pages as recited in claim 2, the at least one server further programmed for:

converting received Web content requests into stored procedures;

maintaining a database query log from the URLs contained in the stored procedures and the queries issued to the least one second memory in response to the stored procedures;

maintaining an operation log from the URLs contained in the stored procedures and the operations issued to the least one second memory in response to the stored procedures;

generating a URL/relevant operation mapping table from the database query log and the operation log; and

determining, from the identity of the modified data, the maintained associations between the issued queries or operations and the stored data, and the URL/relevant operation mapping table which stored Web pages are associated with the identified modified data.

39. A system for updating stored Web pages as recited in claim 38, the at least one server including:

an application server programmed for converting received Web content requests into stored procedures, generating the database query log from the URLs contained in the stored procedures and the queries issued to the least one second memory in response to the stored procedures, and generating the operation log from the URLs contained in the stored procedures and the operations issued to the least one second memory in response to the stored procedures;

a content change monitor programmed for maintaining the associations between the issued queries or operations and the stored data accessed by the queries or operations, and determining which queries or operations are associated with the identified modified data; and

a content delivery services server for generating a URL/relevant operation mapping table from the database query log and the operation log and determining, from the identity of the modified data, the maintained associations between the issued queries or operations and the stored data, and the URL/relevant operation mapping table, which stored Web pages are associated with the identified modified data, and communicating the update command to the at least one first memory that contains the stored Web pages associated with the identified modified data.

40. A system for updating the stored Web pages as recited in claim 39, the content change monitor programmed for:

receiving queries sent to the at least one second memory;

defining a view table for each query containing pointers to all data in the at least one second memory that is associated with that particular query;

receiving the identified modified data from the memory management system; and

determining from the view table and the identified modified data any queries associated with the identified modified data.

41. A system for updating the stored Web pages as recited in claim 39, the content change monitor programmed for:

receiving queries or operations sent to the at least one second memory;

defining a daemon for each query or operation, the daemon establishing criteria for identifying stored data in the at least one second memory that is associated with that particular query or operation; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the daemon.

42. A system for updating the stored Web pages as recited in claim 39, the content change monitor programmed for:

receiving queries sent to the at least one second memory;

creating a trigger definition for each query, the trigger definition establishing criteria for identifying stored data in the at least one second memory associated with that particular query; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the trigger definition.

43. A system for updating the stored Web pages as recited in claim 39, the content change monitor containing at least one third memory and further programmed for:

replicating data from the at least one second memory to the at least one third memory to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

44. A system for updating the stored Web pages as recited in claim 40, the content change monitor containing at least one third memory and further programmed for:

performing all operations in the database log on the at least one third memory, in sequential order, to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

45. A system for updating the stored Web pages as recited in claim 38, further including a sniffer for listening to network traffic by reading request/delivery logs, collecting URL information and query or operation instances by reading a query instance delivery log, and generating the associations between the URL of the Web content request and the issued queries or operations.

46. A system for updating the stored Web pages as recited in claim 38, further including an invalidator/refresher for notifying appropriate caches or Web servers of the invalidity of a particular stored Web page, the invalidator/refresher including:

a registration module for creating invalidation policies and collecting relevant query or operation statistics;

an invalidation module for reading the database update logs and the associations between the URL of the Web content request and the issued queries or operations, scheduling subqueries, and notifying appropriate caches or Web servers of the invalidity of a particular stored Web page; and

an information management module for creating auxiliary data structures for the invalidation module.

47. A system for updating the stored Web pages as recited in claim 38, the at least one server further programmed for:

receiving a query sent to the at least one second memory, the query including one or more query conditions linked by operators, the linked query conditions identifying data in the at least one second memory associated with that query; and

determining if the query is associated with the identified modified data by determining if the identified modified data satisfies the linked query conditions.

48. A system for updating the stored Web pages as recited in claim 47, wherein if the linked query conditions require access to more than one table in the at least one second memory, the at least one server is further programmed such that for each table to which access is required, the at least one server determines if the identified modified data satisfies those portions of the linked query condition specific to that particular table;

wherein after it is determined whether the identified modified data satisfies those portions of the linked query condition specific to a particular table, those portions of the linked query condition specific to a particular table are replaced by a determined value prior to evaluating those portions of the linked query condition specific to another particular table; and

wherein, if at any time during the determination of whether the identified modified data satisfies those portions of the linked query condition specific to a particular table, it is determined that the linked query conditions are satisfied, then the identified modified data satisfies the linked query conditions.

49. A system for updating the stored Web pages as recited in claim 48, wherein after those portions of the linked query condition specific to a particular table are evaluated and a value for those portions is determined, in order to evaluate those portions of the linked query condition specific to another particular table, the at least one server is further programmed for:

issuing a subquery to the other particular table containing the linked query conditions, with portions of the linked query condition replaced by any previously determined values.

50. A system for updating the stored Web pages as recited in claim 49, wherein after a subquery issued to a particular table returns determined values for that subquery, the determined values are stored in an index structure;

wherein if subsequent queries are received by the at least one server that would require reissuing the same subquery, the index structure may be accessed by the at least one server to avoid having to issue the same subquery.

51. A system for updating stored Web pages as recited in claim 2, the at least one server further programmed for:

extracting URL information from the received the Web content requests;

extracting query and operation information from the issued queries and operations;

maintaining a URL/relevant operation mapping log from the extracted URL information and the extracted query and operation information; and

determining, from the identity of the modified data, the maintained associations between the issued queries or operations and the stored data, and the URL/relevant operation mapping log, which stored Web pages are associated with the identified modified data.

52. A system for updating stored Web pages as recited in claim 51, the at least one server including:

an application server proxy for extracting URL information from the received the Web content requests, extracting query and operation information from the issued queries and operations, forwarding the received Web content requests, and maintaining the URL/relevant operation mapping log;

an application server for receiving Web server requests forwarded by the application server proxy and issuing the at least one query or operation in response to the forwarded Web content request;

a content change monitor programmed for maintaining the associations between the issued queries or operations and the stored data accessed by the queries or operations, and determining which queries or operations are associated with the identified modified data; and

a content delivery services server programmed for receiving an identity of the queries or operations associated with the identified modified data from the content change monitor, receiving the URL/relevant operation mapping log, determining which stored Web pages are associated with the queries or operations associated with the identified modified data, and communicating the update command to the at least one first memory that contains the stored Web pages associated with the identified modified data.

53. A system for updating the stored Web pages as recited in claim 52, the content change monitor programmed for:

receiving queries sent to the at least one second memory;

defining a view table for each query containing pointers to all data in the at least one second memory that is associated with that particular query;

receiving the identified modified data from the memory management system; and

determining from the view table and the identified modified data any queries associated with the identified modified data.

54. A system for updating the stored Web pages as recited in claim 52, the content change monitor programmed for:

receiving operations sent to the at least one second memory;

defining a daemon for each operation, the daemon establishing criteria for identifying stored data in the at least one second memory that is associated with that particular operation; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the daemon.

55. A system for updating the stored Web pages as recited in claim 52, the content change monitor programmed for:

receiving queries sent to the at least one second memory;

creating a trigger definition for each query, the trigger definition establishing criteria for identifying stored data in the at least one second memory associated with that particular query; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the trigger definition.

56. A system for updating the stored Web pages as recited in claim 52, the content change monitor containing at least one third memory and further programmed for:

replicating data from the at least one second memory to the at least one third memory to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

57. A system for updating the stored Web pages as recited in claim 53, the content change monitor containing at least one third memory and further programmed for:

performing all operations in the database log on the at least one third memory, in sequential order, to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

58. A system for updating the stored Web pages as recited in claim 51, further including a sniffer for listening to network traffic by reading request/delivery logs, collecting URL information and query or operation instances by reading a query instance delivery log, and generating the associations between the URL of the Web content request and the issued queries or operations.

59. A system for updating the stored Web pages as recited in claim 51, further including an invalidator/refresher for notifying appropriate caches or Web servers of the invalidity of a particular stored Web page, the invalidator/refresher including:

a registration module for creating invalidation policies and collecting relevant query or operation statistics;

an invalidation module for reading the database update logs and the associations between the URL of the Web content request and the issued queries or operations, scheduling subqueries, and notifying appropriate caches or Web servers of the invalidity of a particular stored Web page; and

an information management module for creating auxiliary data structures for the invalidation module.

60. A system for updating the stored Web pages as recited in claim 51, the at least one server further programmed for:

receiving a query sent to the at least one second memory, the query including one or more query conditions linked by operators, the linked query conditions identifying data in the at least one second memory associated with that query; and

determining if the query is associated with the identified modified data by determining if the identified modified data satisfies the linked query conditions.

61. A system for updating the stored Web pages as recited in claim 60, wherein if the linked query conditions require access to more than one table in the at least one second memory, the at least one server is further programmed such that for each table to which access is required, the at least one server determines if the identified modified data satisfies those portions of the linked query condition specific to that particular table;

wherein after it is determined whether the identified modified data satisfies those portions of the linked query condition specific to a particular table, those portions of the linked query condition specific to a particular table are replaced by a determined value prior to evaluating those portions of the linked query condition specific to another particular table; and

wherein, if at any time during the determination of whether the identified modified data satisfies those portions of the linked query condition specific to a particular table, it is determined that the linked query conditions are satisfied, then the identified modified data satisfies the linked query conditions.

62. A system for updating the stored Web pages as recited in claim 61, wherein after those portions of the linked query condition specific to a particular table are evaluated and a value for those portions is determined, in order to evaluate those portions of the linked query condition specific to another particular table, the at least one server is further programmed for:

issuing a subquery to the other particular table containing the linked query conditions, with portions of the linked query condition replaced by any previously determined values.

63. A system for updating the stored Web pages as recited in claim 62, wherein after a subquery issued to a particular table returns determined values for that subquery, the determined values are stored in an index structure;

wherein if subsequent queries are received by the at least one server that would require reissuing the same subquery, the index structure may be accessed by the at least one server to avoid having to issue the same subquery.

64. In a system having at least one first memory for storing Web pages and a memory management system including at least one second memory for storing data used to generate Web pages, the memory management system capable of identifying modified data stored in the at least one second memory and maintaining at least one database log sequentially containing all operations to the at least one second memory, a method for updating stored Web pages based on modifications to data stored in the at least one second memory comprising the steps of:

maintaining associations between each stored Web page and the stored data used to generate that particular Web page,

receiving the identity of modified data from the memory management system,

determining, from the identified modified data and the maintained associations, which stored Web pages are associated with the identified modified data, and

communicating an update command to the at least one first memory that contains the stored Web pages associated with the identified modified data for updating the stored Web pages.

65. A method for updating stored Web pages as recited in claim 64, the method further including the steps of:

receiving a Web content request containing a URL;

issuing at least one query or operation to the at least one second memory in response to the Web content request;

maintaining associations between the URL of the Web content request and the issued queries or operations;

maintaining associations between the issued queries or operations and the stored data accessed by the queries or operations;

deriving associations between the stored Web pages and the stored data from the maintained associations between the URL of the Web content request and the issued queries or operations, and the maintained associations between the issued queries or operations and the stored data accessed by the queries or operations; and

identifying, through prior processing steps to store the stored Web pages in the at least one first memory, the location of the Web pages associated with the identified modified data.

66. A method for updating the stored Web pages as recited in claim 65, wherein the step of communicating an update command includes communicating a message to invalidate or delete the stored Web pages.

67. A method for updating the stored Web pages as recited in claim 65, wherein the step of communicating an update command includes communicating a message to refresh the stored Web pages.

68. A method for updating the stored Web pages as recited in claim 67, wherein if the stored Web page is in a Web server, the step of communicating an update command includes communicating a message to generate a new Web page and store the new Web page in the Web server.

69. A method for updating stored Web pages as recited in claim 65, the method further including the steps of:

maintaining a Web server log containing a request and delivery timestamp for the received URL;

maintaining a query and operation log containing an issuance timestamp for the issued query or operation;

deriving associations between the stored Web pages and the stored data from the Web server log and the query and operation log; and

determining, from the identity of the modified data and the derived associations between the stored Web pages and the stored data, which stored Web pages are associated with the identified modified data.

70. A method for updating the stored Web pages as recited in claim 69, the method further including the steps of:

receiving queries sent to the at least one second memory;

defining a view table for each query containing pointers to all data in the at least one second memory that is associated with that particular query;

receiving the identified modified data from the memory management system; and

determining from the view table and the identified modified data any queries associated with the identified modified data.

71. A method for updating the stored Web pages as recited in claim 69, the method further including the steps of:

receiving operations sent to the at least one second memory;

defining a daemon for each operation, the daemon establishing criteria for identifying stored data in the at least one second memory that is associated with that particular operation; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the daemon.

72. A method for updating the stored Web pages as recited in claim 69, the method further including the steps of:

receiving queries sent to the at least one second memory;

creating a trigger definition for each query, the trigger definition establishing criteria for identifying stored data in the at least one second memory associated with that particular query; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the trigger definition.

73. A method for updating the stored Web pages as recited in claim 70, the method further including the steps of:

replicating data from the at least one second memory to at least one third memory to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

74. A method for updating the stored Web pages as recited in claim 70, the method further including the steps of:

performing all operations in a database log on at least one third memory, in sequential order, to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

75. A method for updating the stored Web pages as recited in claim 69, the method further including the steps of listening to network traffic by reading request/delivery logs, collecting URL information and query or operation instances by reading a query instance delivery log, and generating the associations between the URL of the Web content request and the issued queries or operations.

76. A method for updating the stored Web pages as recited in claim 69, the method further including the steps of:

creating invalidation policies and collecting relevant query or operation statistics;

reading database update logs and the associations between the URL of the Web content request and the issued queries or operations, scheduling subqueries, and notifying appropriate caches or Web servers of the invalidity of a particular stored Web page; and

creating auxiliary data structures.

77. A method for updating the stored Web pages as recited in claim 69, the method further including the steps of:

receiving a query sent to the at least one second memory, the query including one or more query conditions linked by operators, the linked query conditions identifying data in the at least one second memory associated with that query; and

determining if the query is associated with the identified modified data by determining if the identified modified data satisfies the linked query conditions.

78. A method for updating the stored Web pages as recited in claim 77, wherein if the linked query conditions require access to more than one table in the at least one second memory, the method further includes the step of determining, for each table to which access is required, if the identified modified data satisfies those portions of the linked query condition specific to that particular table;

wherein after it is determined whether the identified modified data satisfies those portions of the linked query condition specific to a particular table, the method further includes the step of replacing those portions of the linked query condition specific to a particular table by a determined value prior to evaluating those portions of the linked query condition specific to another particular table; and

wherein the method further includes the step of determining that the identified modified data satisfies the linked query conditions if, at any time, the linked query conditions are satisfied.

79. A method for updating the stored Web pages as recited in claim 78, wherein after those portions of the linked query condition specific to a particular table are evaluated and a value for those portions is determined, in order to evaluate those portions of the linked query condition specific to another particular table, the method further includes the step of:

issuing a subquery to the other particular table containing the linked query conditions, with portions of the linked query condition replaced by any previously determined values.

80. A method for updating the stored Web pages as recited in claim 79, wherein after a subquery issued to a particular table returns determined values for that subquery, the method further includes the step of storing the determined values in an index structure;

wherein if subsequent queries are received by the at least one server that would require reissuing the same subquery, the method further includes the step of accessing the index structure to avoid having to issue the same subquery.

81. A method for updating stored Web pages as recited in claim 65, the method further including the steps of:

maintaining a URL/relevant operation mapping log from the URLs contained in the received Web content requests and the queries or operations issued to the least one second memory in response to the Web content requests; and

determining, from the identity of the modified data, the maintained associations between the issued queries or operations and the stored data, and the URL/relevant operation mapping log, which stored Web pages are associated with the identified modified data.

82. A method for updating the stored Web pages as recited in claim 81, the method further including the steps of:

receiving queries sent to the at least one second memory;

defining a view table for each query containing pointers to all data in the at least one second memory that is associated with that particular query;

receiving the identified modified data from the memory management system; and

determining from the view table and the identified modified data any queries associated with the identified modified data.

83. A method for updating the stored Web pages as recited in claim 81, the method further including the steps of:

receiving operations sent to the at least one second memory;

defining a daemon for each operation, the daemon establishing criteria for identifying stored data in the at least one second memory that is associated with that particular operation; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the daemon.

84. A method for updating the stored Web pages as recited in claim 81, the method further including the steps of:

receiving queries sent to the at least one second memory;

creating a trigger definition for each query, the trigger definition establishing criteria for identifying stored data in the at least one second memory associated with that particular query; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the trigger definition.

85. A method for updating the stored Web pages as recited in claim 82, the method further including the steps of:

replicating data from the at least one second memory to at least one third memory to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

86. A method for updating the stored Web pages as recited in claim 82, the method further including the steps of:

performing all operations in the database log on at least one third memory, in sequential order, to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

87. A method for updating the stored Web pages as recited in claim 81, the method further including the steps of listening to network traffic by reading request/delivery logs, collecting URL information and query or operation instances by reading a query instance delivery log, and generating the associations between the URL of the Web content request and the issued queries or operations.

88. A method for updating the stored Web pages as recited in claim 81, the method further including the steps of:

creating invalidation policies and collecting relevant query or operation statistics;

reading database update logs and the associations between the URL of the Web content request and the issued queries or operations, scheduling subqueries, and notifying appropriate caches or Web servers of the invalidity of a particular stored Web page; and

creating auxiliary data structures.

89. A method for updating the stored Web pages as recited in claim 81, the method further including the steps of:

receiving a query sent to the at least one second memory, the query including one or more query conditions linked by operators, the linked query conditions identifying data in the at least one second memory associated with that query; and

determining if the query is associated with the identified modified data by determining if the identified modified data satisfies the linked query conditions.

90. A method for updating the stored Web pages as recited in claim 89, wherein if the linked query conditions require access to more than one table in the at least one second memory, the method further includes the step of determining, for each table to which access is required, if the identified modified data satisfies those portions of the linked query condition specific to that particular table;

wherein after it is determined whether the identified modified data satisfies those portions of the linked query condition specific to a particular table, the method further includes the step of replacing those portions of the linked query condition specific to a particular table by a determined value prior to evaluating those portions of the linked query condition specific to another particular table; and

wherein the method further including the step of determining that the identified modified data satisfies the linked query conditions if, at any time, the linked query conditions are satisfied.

91. A method for updating the stored Web pages as recited in claim 90, wherein after those portions of the linked query condition specific to a particular table are evaluated and a value for those portions is determined, in order to evaluate those portions of the linked query condition specific to another particular table, the method further includes the step of:

issuing a subquery to the other particular table containing the linked query conditions, with portions of the linked query condition replaced by any previously determined values.

92. A method for updating the stored Web pages as recited in claim 91, wherein after a subquery issued to a particular table returns determined values for that subquery, the method further includes the step of storing the determined values in an index structure;

wherein if subsequent queries are received by the at least one server that would require reissuing the same subquery, the method further including the step of accessing the index structure to avoid having to issue the same subquery.

93. A method for updating stored Web pages as recited in claim 65, the method further including the steps of:

converting received Web content requests into stored procedures;

maintaining a database query log from the URLs contained in the stored procedures and the queries issued to the least one second memory in response to the stored procedures;

maintaining an operation log from the URLs contained in the stored procedures and the operations issued to the least one second memory in response to the stored procedures;

generating a URL/relevant operation mapping table from the database query log and the operation log; and

determining, from the identity of the modified data, the maintained associations between the issued queries or operations and the stored data, and the URL/relevant operation mapping table which stored Web pages are associated with the identified modified data.

94. A method for updating the stored Web pages as recited in claim 93, the method further including the steps of:

receiving queries sent to the at least one second memory;

defining a view table for each query containing pointers to all data in the at least one second memory that is associated with that particular query;

receiving the identified modified data from the memory management system; and

determining from the view table and the identified modified data any queries associated with the identified modified data.

95. A method for updating the stored Web pages as recited in claim 93, the method further including the steps of:

receiving queries or operations sent to the at least one second memory;

defining a daemon for each query or operation, the daemon establishing criteria for identifying stored data in the at least one second memory that is associated with that particular query or operation; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the daemon.

96. A method for updating the stored Web pages as recited in claim 93, the method further including the steps of:

receiving queries sent to the at least one second memory;

creating a trigger definition for each query, the trigger definition establishing criteria for identifying stored data in the at least one second memory associated with that particular query; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the trigger definition.

97. A method for updating the stored Web pages as recited in claim 94, the method further including the steps of:

replicating data from the at least one second memory to at least one third memory to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

98. A method for updating the stored Web pages as recited in claim 94, the method further including the steps of:

performing all operations in the database log on at least one third memory, in sequential order, to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

99. A method for updating the stored Web pages as recited in claim 93, the method further including the steps of listening to network traffic by reading request/delivery logs, collecting URL information and query or operation instances by reading a query instance delivery log, and generating the associations between the URL of the Web content request and the issued queries or operations.

100. A method for updating the stored Web pages as recited in claim 93, the method further including the steps of:

creating invalidation policies and collecting relevant query or operation statistics;

reading database update logs and the associations between the URL of the Web content request and the issued queries or operations, scheduling subqueries, and notifying appropriate caches or Web servers of the invalidity of a particular stored Web page; and

creating auxiliary data structures.

101. A method for updating the stored Web pages as recited in claim 93, the method further including the steps of:

receiving a query sent to the at least one second memory, the query including one or more query conditions linked by operators, the linked query conditions identifying data in the at least one second memory associated with that query; and

determining if the query is associated with the identified modified data by determining if the identified modified data satisfies the linked query conditions.

102. A method for updating the stored Web pages as recited in claim 101, wherein if the linked query conditions require access to more than one table in the at least one second memory, the method further includes the step of determining, for each table to which access is required, if the identified modified data satisfies those portions of the linked query condition specific to that particular table;

wherein after it is determined whether the identified modified data satisfies those portions of the linked query condition specific to a particular table, the method further includes the step of replacing those portions of the linked query condition specific to a particular table by a determined value prior to evaluating those portions of the linked query condition specific to another particular table; and

wherein the method further includes the step of determining that the identified modified data satisfies the linked query conditions if, at any time, the linked query conditions are satisfied.

103. A method for updating the stored Web pages as recited in claim 102, wherein after those portions of the linked query condition specific to a particular table are evaluated and a value for those portions is determined, in order to evaluate those portions of the linked query condition specific to another particular table, the method further includes the step of:

issuing a subquery to the other particular table containing the linked query conditions, with portions of the linked query condition replaced by any previously determined values.

104. A method for updating the stored Web pages as recited in claim 103, wherein after a subquery issued to a particular table returns determined values for that subquery, the method further includes the step of storing the determined values in an index structure;

wherein if subsequent queries are received by the at least one server that would require reissuing the same subquery, the method further includes the step of accessing the index structure to avoid having to issue the same subquery.

105. A method for updating stored Web pages as recited in claim 65, the method further including the steps of:

extracting URL information from the received the Web content requests;

extracting query and operation information from the issued queries and operations;

maintaining a URL/relevant operation mapping log from the extracted URL information and the extracted query and operation information; and

determining, from the identity of the modified data, the maintained associations between the issued queries or operations and the stored data, and the URL/relevant operation mapping log, which stored Web pages are associated with the identified modified data.

106. A method for updating the stored Web pages as recited in claim 105, the method further including the steps of:

receiving queries sent to the at least one second memory;

defining a view table for each query containing pointers to all data in the at least one second memory that is associated with that particular query;

receiving the identified modified data from the memory management system; and

determining from the view table and the identified modified data any queries associated with the identified modified data.

107. A method for updating the stored Web pages as recited in claim 105, the method further including the steps of:

receiving operations sent to the at least one second memory;

defining a daemon for each operation, the daemon establishing criteria for identifying stored data in the at least one second memory that is associated with that particular operation; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the daemon.

108. A method for updating the stored Web pages as recited in claim 105, the method further including the steps of:

receiving queries sent to the at least one second memory;

creating a trigger definition for each query, the trigger definition establishing criteria for identifying stored data in the at least one second memory associated with that particular query; and

determining queries associated with the identified modified data by determining if the identified modified data satisfies the trigger definition.

109. A method for updating the stored Web pages as recited in claim 106, the method further including the steps of:

replicating data from the at least one second memory to at least one third memory to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

110. A method for updating the stored Web pages as recited in claim 106, the method further including the steps of:

performing all operations in the database log on at least one third memory, in sequential order, to harmonize the at least one third memory with the at least one second memory; and

defining the view table for each query containing pointers to all data in the at least one third memory that is associated with that particular query.

111. A method for updating the stored Web pages as recited in claim 105, the method further including the steps of listening to network traffic by reading request/delivery logs, collecting URL information and query or operation instances by reading a query instance delivery log, and generating the associations between the URL of the Web content request and the issued queries or operations.

112. A method for updating the stored Web pages as recited in claim 105, the method further including the steps of:

creating invalidation policies and collecting relevant query or operation statistics;

reading database update logs and the associations between the URL of the Web content request and the issued queries or operations, scheduling subqueries, and notifying appropriate caches or Web servers of the invalidity of a particular stored Web page; and

creating auxiliary data structures.

113. A method for updating the stored Web pages as recited in claim 105, the method further including the steps of:

receiving a query sent to the at least one second memory, the query including one or more query conditions linked by operators, the linked query conditions identifying data in the at least one second memory associated with that query; and

determining if the query is associated with the identified modified data by determining if the identified modified data satisfies the linked query conditions.

114. A method for updating the stored Web pages as recited in claim 113, wherein if the linked query conditions require access to more than one table in the at least one second memory, the method further includes the step of determining, for each table to which access is required, if the identified modified data satisfies those portions of the linked query condition specific to that particular table;

wherein after it is determined whether the identified modified data satisfies those portions of the linked query condition specific to a particular table, the method further includes the step of replacing those portions of the linked query condition specific to a particular table by a determined value prior to evaluating those portions of the linked query condition specific to another particular table; and

wherein the method further includes the step determining that the identified modified data satisfies the linked query conditions if, at any time, the linked query conditions are satisfied.

115. A method for updating the stored Web pages as recited in claim 114, wherein after those portions of the linked query condition specific to a particular table are evaluated and a value for those portions is determined, in order to evaluate those portions of the linked query condition specific to another particular table, the method further includes the step of:

issuing a subquery to the other particular table containing the linked query conditions, with portions of the linked query condition replaced by any previously determined values.

116. A method for updating the stored Web pages as recited in claim 115, wherein after a subquery issued to a particular table returns determined values for that subquery, the method further includes the step of storing the determined values in an index structure;

wherein if subsequent queries are received by the at least one server that would require reissuing the same subquery, the method further includes the step of accessing the index structure to avoid having to issue the same subquery.

117. A system for updating the stored Web pages as recited in claim 14, the registration module further programmed for:

registering, in an off-line mode, those query types that the invalidator/refresher must detect, and storing the registered query types in an auxiliary data structure;

scanning the associations between the URL of the Web content request and the issued queries or operations, locating query instances that cannot be associated with a known query type, interpreting those query instances to identify their query types, and registering, in an on-line mode, the identified query types; and

determining, based on join index size, update frequency, or query frequency, whether a join index should be maintained;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

118. A system for updating the stored Web pages as recited in claim 117, wherein the determination of whether a join index should be maintained is favored if the join index size is small, the join index update frequency is low, the database or external data source query frequency is high, or the database or external data source query cost is high.

119. A system for updating the stored Web pages as recited in claim 14, the invalidation module further programmed for:

performing database polling request scheduling by reading invalidation policies and the auxiliary data structures to identify which database polling requests must be generated, and determining when these requests should be sent to memory management systems or external data sources; and

converting database polling requests into subqueries or suboperations readable by the memory management systems or the external data sources;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

120. A system for updating the stored Web pages as recited in claim 14, the information management module further programmed for:

performing query instance coalescing by identifying related query instances and processing them as a group;

performing query type coalescing by identifying related query types and processing them in a coordinated manner; and

performing update coalescing by identifying related database or external data source updates, deletions, or insertions, and processing them as a group.

121. A system for updating the stored Web pages as recited in claim 14, the invalidator/refresher further programmed for:

forming a positive query lattice for a particular query, the positive query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is not satisfied, it can be determined that the database or external data source change will not affect the particular query, and no further query conditions will need to be evaluated;

forming a negative query lattice for a particular query, the negative query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is satisfied, it can be determined that the database or external data source change will affect the particular query, and no further query conditions will need to be evaluated; and

utilizing the positive query lattice for determining when a database or external data source change will not affect a query, and utilizing the negative query lattice for determining when a database or external data source change will affect a query.

122. A system for updating the stored Web pages as recited in claim 14, the invalidator/refresher further programmed for:

determining, based on a number or complexity of queries generated by a particular Web page request, whether to cache, invalidate, or refresh the stored Web pages.

123. A system for updating the stored Web pages as recited in claim 27, the registration module further programmed for:

registering, in an off-line mode, those query types that the invalidator/refresher must detect, and storing the registered query types in an auxiliary data structure;

scanning the associations between the URL of the Web content request and the issued queries or operations, locating query instances that cannot be associated with a known query type, interpreting those query instances to identify their query types, and registering, in an on-line mode, the identified query types; and

determining, based on join index size, update frequency, or query frequency, whether a join index should be maintained;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

124. A system for updating the stored Web pages as recited in claim 123, wherein the determination of whether a join index should be maintained is favored if the join index size is small, the join index update frequency is low, the database or external data source query frequency is high, or the database or external data source query cost is high.

125. A system for updating the stored Web pages as recited in claim 27, the invalidation module further programmed for:

performing database polling request scheduling by reading invalidation policies and the auxiliary data structures to identify which database polling requests must be generated, and determining when these requests should be sent to memory management systems or external data sources; and

converting database polling requests into subqueries or suboperations readable by the memory management systems or the external data sources;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

126. A system for updating the stored Web pages as recited in claim 27, the information management module further programmed for:

performing query instance coalescing by identifying related query instances and processing them as a group;

performing query type coalescing by identifying related query types and processing them in a coordinated manner; and

performing update coalescing by identifying related database or external data source updates, deletions, or insertions, and processing them as a group.

127. A system for updating the stored Web pages as recited in claim 27, the invalidator/refresher further programmed for:

forming a positive query lattice for a particular query, the positive query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is not satisfied, it can be determined that the database or external data source change will not affect the particular query, and no further query conditions will need to be evaluated;

forming a negative query lattice for a particular query, the negative query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is satisfied, it can be determined that the database or external data source change will affect the particular query, and no further query conditions will need to be evaluated; and

utilizing the positive query lattice for determining when a database or external data source change will not affect a query, and utilizing the negative query lattice for determining when a database or external data source change will affect a query.

128. A system for updating the stored Web pages as recited in claim 27, the invalidator/refresher further programmed for:

determining, based on a number or complexity of queries generated by a particular Web page request, whether to cache, invalidate, or refresh the stored Web pages.

129. A system for updating the stored Web pages as recited in claim 46, the registration module further programmed for:

registering, in an off-line mode, those query types that the invalidator/refresher must detect, and storing the registered query types in an auxiliary data structure;

scanning the associations between the URL of the Web content request and the issued queries or operations, locating query instances that cannot be associated with a known query type, interpreting those query instances to identify their query types, and registering, in an on-line mode, the identified query types; and

determining, based on join index size, update frequency, or query frequency, whether a join index should be maintained;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

130. A system for updating the stored Web pages as recited in claim 129, wherein the determination of whether a join index should be maintained is favored if the join index size is small, the join index update frequency is low, the database or external data source query frequency is high, or the database or external data source query cost is high.

131. A system for updating the stored Web pages as recited in claim 46, the invalidation module further programmed for:

performing database polling request scheduling by reading invalidation policies and the auxiliary data structures to identify which database polling requests must be generated, and determining when these requests should be sent to memory management systems or external data sources; and

converting database polling requests into subqueries or suboperations readable by the memory management systems or the external data sources;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

132. A system for updating the stored Web pages as recited in claim 46, the information management module further programmed for:

performing query instance coalescing by identifying related query instances and processing them as a group;

performing query type coalescing by identifying related query types and processing them in a coordinated manner; and

performing update coalescing by identifying related database or external data source updates, deletions, or insertions, and processing them as a group.

133. A system for updating the stored Web pages as recited in claim 46, the invalidator/refresher further programmed for:

forming a positive query lattice for a particular query, the positive query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is not satisfied, it can be determined that the database or external data source change will not affect the particular query, and no further query conditions will need to be evaluated;

forming a negative query lattice for a particular query, the negative query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is satisfied, it can be determined that the database or external data source change will affect the particular query, and no further query conditions will need to be evaluated; and

utilizing the positive query lattice for determining when a database or external data source change will not affect a query, and utilizing the negative query lattice for determining when a database or external data source change will affect a query.

134. A system for updating the stored Web pages as recited in claim 46, the invalidator/refresher further programmed for:

determining, based on a number or complexity of queries generated by a particular Web page request, whether to cache, invalidate, or refresh the stored Web pages.

135. A system for updating the stored Web pages as recited in claim 59, the registration module further programmed for:

registering, in an off-line mode, those query types that the invalidator/refresher must detect, and storing the registered query types in an auxiliary data structure;

scanning the associations between the URL of the Web content request and the issued queries or operations, locating query instances that cannot be associated with a known query type, interpreting those query instances to identify their query types, and registering, in an on-line mode, the identified query types; and

determining, based on join index size, update frequency, or query frequency, whether a join index should be maintained;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

136. A system for updating the stored Web pages as recited in claim 135, wherein the determination of whether a join index should be maintained is favored if the join index size is small, the join index update frequency is low, the database or external data source query frequency is high, or the database or external data source query cost is high.

137. A system for updating the stored Web pages as recited in claim 59, the invalidation module further programmed for:

performing database polling request scheduling by reading invalidation policies and the auxiliary data structures to identify which database polling requests must be generated, and determining when these requests should be sent to memory management systems or external data sources; and

converting database polling requests into subqueries or suboperations readable by the memory management systems or the external data sources;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

138. A system for updating the stored Web pages as recited in claim 59, the information management module further programmed for:

performing query instance coalescing by identifying related query instances and processing them as a group;

performing query type coalescing by identifying related query types and processing them in a coordinated manner; and

performing update coalescing by identifying related database or external data source updates, deletions, or insertions, and processing them as a group.

139. A system for updating the stored Web pages as recited in claim 59, the invalidator/refresher further programmed for:

forming a positive query lattice for a particular query, the positive query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is not satisfied, it can be determined that the database or external data source change will not affect the particular query, and no further query conditions will need to be evaluated;

forming a negative query lattice for a particular query, the negative query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is satisfied, it can be determined that the database or external data source change will affect the particular query, and no further query conditions will need to be evaluated; and

utilizing the positive query lattice for determining when a database or external data source change will not affect a query, and utilizing the negative query lattice for determining when a database or external data source change will affect a query.

140. A system for updating the stored Web pages as recited in claim 59, the invalidator/refresher further programmed for:

determining, based on a number or complexity of queries generated by a particular Web page request, whether to cache, invalidate, or refresh the stored Web pages.

141. A method for updating the stored Web pages as recited in claim 76, the method further including the steps of:

registering, in an off-line mode, those query types that the invalidator/refresher must detect, and storing the registered query types in an auxiliary data structure;

scanning the associations between the URL of the Web content request and the issued queries or operations, locating query instances that cannot be associated with a known query type, interpreting those query instances to identify their query types, and registering, in an on-line mode, the identified query types; and

determining, based on join index size, update frequency, or query frequency, whether a join index should be maintained.

142. A method for updating the stored Web pages as recited in claim 141, wherein the step of determining whether a join index should be maintained comprises favoring the maintenance of a join index if the join index size is small, the join index update frequency is low, the database or external data source query frequency is high, or the database or external data source query cost is high.

143. A method for updating the stored Web pages as recited in claim 76, the method further including the steps of:

performing database polling request scheduling by reading invalidation policies and the auxiliary data structures to identify which database polling requests must be generated, and determining when these requests should be sent to memory management systems or external data sources; and

converting database polling requests into subqueries or suboperations readable by the memory management systems or the external data sources;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

144. A method for updating the stored Web pages as recited in claim 76, the method further including the steps of:

performing query instance coalescing by identifying related query instances and processing them as a group;

performing query type coalescing by identifying related query types and processing them in a coordinated manner; and

performing update coalescing by identifying related database or external data source updates, deletions, or insertions, and processing them as a group.

145. A method for updating the stored Web pages as recited in claim 76, the method further including the steps of:

forming a positive query lattice for a particular query, the positive query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is not satisfied, it can be determined that the database or external data source change will not affect the particular query, and no further query conditions will need to be evaluated;

forming a negative query lattice for a particular query, the negative query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is satisfied, it can be determined that the database or external data source change will affect the particular query, and no further query conditions will need to be evaluated; and

utilizing the positive query lattice for determining when a database or external data source change will not affect a query, and utilizing the negative query lattice for determining when a database or external data source change will affect a query.

146. A method for updating the stored Web pages as recited in claim 76, the method further including the step of determining, based on a number or complexity of queries generated by a particular Web page request, whether to cache, invalidate or refresh the stored Web pages.

147. A method for updating the stored Web pages as recited in claim 88, the method further including the steps of:

registering, in an off-line mode, those query types that the invalidator/refresher must detect, and storing the registered query types in an auxiliary data structure;

scanning the associations between the URL of the Web content request and the issued queries or operations, locating query instances that cannot be associated with a known query type, interpreting those query instances to identify their query types, and registering, in an on-line mode, the identified query types; and

determining, based on join index size, update frequency, or query frequency, whether a join index should be maintained;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

148. A method for updating the stored Web pages as recited in claim 147, wherein the step of determining whether a join index should be maintained comprises favoring the maintenance of a join index if the join index size is small, the join index update frequency is low, the database or external data source query frequency is high, or the database or external data source query cost is high.

149. A method for updating the stored Web pages as recited in claim 88, the method further including the steps of:

performing database polling request scheduling by reading invalidation policies and the auxiliary data structures to identify which database polling requests must be generated, and determining when these requests should be sent to memory management systems or external data sources; and

converting database polling requests into subqueries or suboperations readable by the memory management systems or the external data sources;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

150. A method for updating the stored Web pages as recited in claim 88, the method further including the steps of:

performing query instance coalescing by identifying related query instances and processing them as a group;

performing query type coalescing by identifying related query types and processing them in a coordinated manner; and

performing update coalescing by identifying related database or external data source updates, deletions, or insertions, and processing them as a group.

151. A method for updating the stored Web pages as recited in claim 88, the method further including the steps of:

forming a positive query lattice for a particular query, the positive query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is not satisfied, it can be determined that the database or external data source change will not affect the particular query, and no further query conditions will need to be evaluated;

forming a negative query lattice for a particular query, the negative query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is satisfied, it can be determined that the database or external data source change will affect the particular query, and no further query conditions will need to be evaluated; and

utilizing the positive query lattice for determining when a database or external data source change will not affect a query, and utilizing the negative query lattice for determining when a database or external data source change will affect a query.

152. A method for updating the stored Web pages as recited in claim 88, the method further including the step of determining, based on a number or complexity of queries generated by a particular Web page request, whether to invalidate or refresh the stored Web pages.

153. A method for updating the stored Web pages as recited in claim 100, the method further including the steps of:

registering, in an off-line mode, those query types that the invalidator/refresher must detect, and storing the registered query types in an auxiliary data structure;

scanning the associations between the URL of the Web content request and the issued queries or operations, locating query instances that cannot be associated with a known query type, interpreting those query instances to identify their query types, and registering, in an on-line mode, the identified query types; and

determining, based on join index size, update frequency, or query frequency, whether a join index should be maintained;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

154. A method for updating the stored Web pages as recited in claim 153, wherein the step of determining whether a join index should be maintained comprises favoring the maintenance of a join index if the join index size is small, the join index update frequency is low, the database or external data source query frequency is high, or the database or external data source query cost is high.

155. A method for updating the stored Web pages as recited in claim 100, the method further including the steps of:

performing database polling request scheduling by reading invalidation policies and the auxiliary data structures to identify which database polling requests must be generated, and determining when these requests should be sent to memory management systems or external data sources; and

converting database polling requests into subqueries or suboperations readable by the memory management systems or the external data sources;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

156. A method for updating the stored Web pages as recited in claim 100, the method further including the steps of:

performing query instance coalescing by identifying related query instances and processing them as a group;

performing query type coalescing by identifying related query types and processing them in a coordinated manner; and

performing update coalescing by identifying related database or external data source updates, deletions, or insertions, and processing them as a group.

157. A method for updating the stored Web pages as recited in claim 100, the method further including the steps of:

forming a positive query lattice for a particular query, the positive query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is not satisfied, it can be determined that the database or external data source change will not affect the particular query, and no further query conditions will need to be evaluated;

forming a negative query lattice for a particular query, the negative query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is satisfied, it can be determined that the database or external data source change will affect the particular query, and no further query conditions will need to be evaluated; and

utilizing the positive query lattice for determining when a database or external data source change will not affect a query, and utilizing the negative query lattice for determining when a database or external data source change will affect a query.

158. A method for updating the stored Web pages as recited in claim 100, the method further including the step of determining, based on a number or complexity of queries generated by a particular Web page request, whether to invalidate or refresh the stored Web pages.

159. A method for updating the stored Web pages as recited in claim 112, the method further including the steps of:

registering, in an off-line mode, those query types that the invalidator/refresher must detect, and storing the registered query types in an auxiliary data structure;

scanning the associations between the URL of the Web content request and the issued queries or operations, locating query instances that cannot be associated with a known query type, interpreting those query instances to identify their query types, and registering, in an on-line mode, the identified query types; and

determining, based on join index size, update frequency, or query frequency, whether a join index should be maintained;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

160. A method for updating the stored Web pages as recited in claim 159, wherein the step of determining whether a join index should be maintained comprises favoring the maintenance of a join index if the join index size is small, the join index update frequency is low, the database or external data source query frequency is high, or the database or external data source query cost is high.

161. A method for updating the stored Web pages as recited in claim 112, the method further including the steps of:

performing database polling request scheduling by reading invalidation policies and the auxiliary data structures to identify which database polling requests must be generated, and determining when these requests should be sent to memory management systems or external data sources; and

converting database polling requests into subqueries or suboperations readable by the memory management systems or the external data sources;

wherein updating the stored Web pages includes refreshing, invalidating, or deleting the stored Web pages.

162. A method for updating the stored Web pages as recited in claim 112, the method further including the steps of:

performing query instance coalescing by identifying related query instances and processing them as a group;

performing query type coalescing by identifying related query types and processing them in a coordinated manner; and

performing update coalescing by identifying related database or external data source updates, deletions, or insertions, and processing them as a group.

163. A method for updating the stored Web pages as recited in claim 112, the method further including the steps of:

forming a positive query lattice for a particular query, the positive query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is not satisfied, it can be determined that the database or external data source change will not affect the particular query, and no further query conditions will need to be evaluated;

forming a negative query lattice for a particular query, the negative query lattice comprised of a sequence of query conditions arranged in an order such that if a database or external data source change is detected, and the sequence of query conditions is evaluated, as soon as one query condition is satisfied, it can be determined that the database or external data source change will affect the particular query, and no further query conditions will need to be evaluated; and

utilizing the positive query lattice for determining when a database or external data source change will not affect a query, and utilizing the negative query lattice for determining when a database or external data source change will affect a query.

164. A method for updating the stored Web pages as recited in claim 112, the method further including the step of determining, based on a number or complexity of queries generated by a particular Web page request, whether to invalidate or refresh the stored Web pages.

Description

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates, generally, to content delivery networks and, in preferred embodiments, to systems and methods for intelligent caching and refreshing of dynamically generated and static Web content to improve content delivery services.

2. Description of the Related Art

Web performance is a key point of differentiation among content providers. Crashes and slowdowns within major Web sites demonstrate the difficulties companies face in trying to deal with high Web traffic. As Internet backbone technologies have developed, many innovations in the area of service management have improved bandwidth and Web content retrieval response time. These improvements to infrastructure, however, cannot solve traffic problems at all points within the Internet.

For example, assume in FIG. 1 that an end user 12 in a network 14 in Japan requests access to a page from a Web site 16 in a network 18 in the United States. The request must pass through several gateways 20, 22, and 24 before reaching Web site 16. Although Web site 16 may have large bandwidth (the ability to rapidly communicate large quantities of data), the gateways connecting the network 14 in Japan to the network 18 in the United States may be slow, and thus, when end user 12 attempts to access the page from Web site 16, the gateways may create a bottleneck. Such gateway bottlenecks may result in the access time for one page of data being on the order of 10 seconds or more. Because of the gateway bottlenecks, and because there are many uncertainties along the Internet paths from end user 12 to/from the Web site 16, content delivery networks or systems are now being developed.

Fundamentally, content delivery systems may be designed and deployed for at least two major purposes; one is to achieve load balancing, and the other is to reduce response time. A content delivery system may be implemented using a high speed dedicated line to deliver content while bypassing all gateways or reducing the number of Internet gateways in the transmission path. However, such a dedicated network is expensive and cannot be deployed for all networks. Another approach to implementing content delivery systems is through the use of intelligent caching, mirroring, proxy servers, or other techniques which redirect end users to available servers that contain the desired content and are close to or easily accessible by the end users to ensure fast response time. With some of the traffic redirected, traffic surges will decrease and end users benefit from faster response time. The term generally used for the architecture and functionality of such networks or systems is content delivery services (CDS).

Many approaches and architectures have been proposed for CDS and most of these services, systems, and architectures have focused on static content. For example, CachePortal.TM. is a system providing CDS, and is described in pending U.S. patent application Ser. No. 09/545,805, entitled "System and Method for Efficient Content Delivery," filed Apr. 7, 2000, the contents of which are incorporated herein by reference. CachePortal.TM. has access to mirror servers which are used as edge cache to provide content to end users with less network delay. CachePortal.TM. can distribute content among mirror servers as well as remove, refresh, or invalidate the content in the mirror servers. CachePortal.TM. can also modify the attributes of content in the mirror servers. For example, CachePortal.TM. may check whether an object has been updated. If CachePortal.TM. finds that it is not updated, then CachePortal.TM. may change the value of the refresh time stamp or last modified date time stamp.

However, for many e-business sites, Web pages are created dynamically based on the current state of business processes represented in application servers and databases. The techniques required are more complex than what are needed for static content delivery. Given the fact that based on current technology, the application servers, databases, and Web servers or cache which deliver Web pages are independent components, there has been no efficient mechanism to reflect database content changes in the cached Web pages. As a result, conventional application servers typically specify dynamically generated Web pages to be non-cacheable or expire immediately. By so doing, the processing time for such requests will be the time required for the network round trip--time to receive the current data from the Web servers, and time for the backend systems (i.e. application servers and databases) to perform the computations necessary to provide current data.

FIG. 2 shows an overview of a typical current Web page delivery mechanism 24 for Web sites with backend systems. For example, an e-business site may utilize a database management system (DBMS) 26 to maintain the price, item descriptions, and quantity of all products the e-business site sells through its Web site. It should be understood that a DBMS, as described herein, includes memory such as a database for storing content. As illustrated in FIG. 2, an end user 28 interacts with a Web browser 30 to access the Internet and requests information 32 on a product at the e-business site. Such a request may include parameters such as the product name and model number, and may include other items such as cookies 34.

Cookies are small files that were sent by a particular Web site, and now reside in the end user's computer. The cookie is used by the Web site to keep track of what the end user is doing, or to identify the user. For example, when an end user clicks on a button at a particular Web site, that Web site may send a cookie to the end user's computer. This cookie may monitor the activity of the end user as the Web site is navigated. The next time that the user logs onto that Web site, the Web site may search for a cookie in a particular location. If it finds the cookie, it will know that the end user is a repeat customer. The Web site may also use the navigational information from the cookie to customize the Web page for that end user.

Referring again to FIG. 2, the Web page request 32 may be sent from the client browser 30 to a cache 36. In FIG. 2, the cache block 36 is identified with dotted lines to indicate that it may be any one of a number of different types of cache. One type of cache is called edge cache. Edge cache is also called proxy cache and is typically managed and maintained by a third party content delivery service provider such as CachePortal.TM.. Content providers can store content into this edge cache and make it available at locations that are close to the end user. Another type of cache is called reverse cache, which is cache that is owned by entities close to the end user and is typically located close to the end user. Another type of cache is called server side cache, which is cache that is owned by the content provider. Yet another type of cache is called Web server cache. Web server cache may be found in machines that typically would deliver Web server content from disk to the end user. However because disk access is slow, the content from the disk may be copied to another location within that Web server. This location would provide faster access to that content than the disk. These storage locations are known as Web server caches.

Referring again to FIG. 2, if the Web page request 32 is not serviced by the cache 36 because the page is not found in cache 36, or is expired or invalidated in cache 36, the request will be passed to the e-business Web server 38. In some e-business applications, some frequently accessed pages, such as catalog pages, are pre-generated in advance and are put on the Web server 38 in advance. For example, a Web server machine 38 may have a sub-directory called HTTPD. Web pages that are static (do not change frequently) may be stored within that sub-directory. If those pages are requested by the client browser 30 and the HTTPD subdirectory contains an up to date copy of the Web page, those pages will be delivered straight from the Web server 38 back to the client browser 30 without ever going to the application server 40.

However, pages such as product information and availability pages are dynamic in that they change frequently. Such pages are typically not stored in the Web server 38, but are dynamically created by backend systems when such a page is requested, which may include application servers 40, DBMS 26, and "filesystem+network" or external data sources 42. When a request 44 for a dynamically created Web page is received, the Web server 38 may utilize a sub-directory called CGI (common gateway interface) to convert the request into a new request 46 (which includes the URL and other parameters) that can be understood by the application server 40. The application server 40 receives the Web page request 46 and other parameters from the Web server 38, performs any necessary computations, and accesses the DBMS 26 by way of queries 240, if necessary. The DBMS 26 then returns the information 46 back to the application server 40, where the application server 40 uses this data to put together the dynamically created Web page 48 in HTML.

It should be understood that the application server 40 may be a different machine from the Web server 38, or it may actually be the same machine as the Web server 38, just performing a different function than the Web server 38. More generally, it should be understood that although the figures in this description indicate separate functional blocks for the Web servers 38, application servers 40, API handlers (see subsequent figures), DBMS 26, and content change monitoring components (see subsequent figures), all of these functional elements may be combined in one server or separated into multiple servers. Furthermore, illustrated in FIG. 2 and several other figures in this description is a functional block entitled Filesystem+Network 42. It should be understood that that this block 42 is a generalization of the DBMS block 26, which may include a DBMS and/or any external source of data.

When the program is executed on the application server 40, it may issue one query 240, get the result 46 back, run for an additional period of time, and then issue subsequent queries 240 to get additional information 46 needed to generate the requested Web page 48. For example, in the airline reservation system example, a first query 240 might be issued to determine the available flights on a particular day, and subsequent queries 240 may be issued to determine the prices for those flights. In so doing, different DBMSs 26 or 42 may need to be accessed. Once the application server 40 has put together the dynamically created Web page 48, it sends the Web page indicated at 48 back to the Web server 38. The Web server 38 then sends a forwarded Web page indicated at 50 back to the Web browser 30.

Additionally, when the Web server 38 sends this dynamically generated page 50 to the Web browser 30, it may also store this page 50 to cache 36. With the dynamically generated page 50 in the cache 36, when another user requests the same information, the Web page 50 can be delivered straight from the cache 36 to the end user, without the additional processing time that would be required if the Web page was to be dynamically created by backend systems.

However, dynamically created Web pages may change often, and thus their storage in cache 36 or in the Web server 38 can be problematic for certain e-business Web sites whose businesses rely on presenting the end user with up-to-date information. However, getting the very latest Web page information may be difficult because the cache 36, Web server 38, application server 40 and the DBMS 26 or 42 may all be located in different machines, and they may be independent entities. Furthermore, there may be little or no coordination between these independently operating machines to ensure that the Web pages stored in cache 36 or in a Web server 38 are up-to-date.

One way to increase the probability that the Web pages in the cache 36 are fresh is to periodically refresh the pages through the Web server 38. However, this may result in many unnecessary requests to the Web server 38, the application server 40, and the DBMSs 26, and even with periodic refreshing, the Web pages in cache 36 cannot be guaranteed to be fresh. Because of the difficulty in ensuring that the data stored in cache 36 is fresh, important dynamic Web pages that are stored in cache 36 are typically set to expire immediately. Alternatively, such Web pages may be designated as non-cacheable.

Thus, a typical request for a dynamic Web page will pass through the Web server 38, application server 40, and DBMS 26 or external data source 42, and then eventually the created Web page will be transmitted back to the end user 28. This network round-trip time includes the time for the backend systems (i.e. application servers 40, DBMSs 26, and external data sources 42) to perform the computations necessary to provide the current data. This time can be in the range of 10-20 seconds, which is very slow. Even though some dynamic Web pages may change only once every hour, the fact that it can change may be enough for businesses to make the content non-cacheable or set for expiration immediately. Thus, even if there are numerous requests prior to any data being changed, conventional systems will force those requests to complete the network round trip, including the execution of processes on backend systems.

SUMMARY OF THE DISCLOSURE

Therefore, it is an advantage of embodiments of the present invention to provide a system and method for intelligent caching and refreshing of Web content to synchronize the data stored in cache or Web servers with the data stored in DBMSs or external data sources.

It is a further advantage of embodiments of the present invention to provide a system and method for intelligent caching and refreshing of Web content where, when data in a DBMS or external data source changes, the Web pages stored in cache or Web servers that utilize that data will be invalidated.

It is a further advantage of embodiments of the present invention to provide a system and method for intelligent caching and refreshing of Web content where, when data in a DBMS or external data source changes, the Web pages stored in Cache or Web servers that utilize that data will be refreshed. As a result, even the dynamically generated pages can be delivered from front end caches or edge caches.

It is a further advantage of embodiments of the present invention to provide a system and method for intelligent caching and refreshing of Web content where, because many content requests can be served from cache, the load to backend systems can be eased.

These and other advantages are accomplished according to a system for updating Web pages stored in cache or Web servers based on modifications to data stored in a DBMS or external data source. Updating may include invalidating or refreshing the Web pages. The system for updating stored Web pages may be part of a larger system having a DBMS for storing data used to generate Web pages. The DBMS is capable of identifying modified data stored in its database.

The system for updating stored Web pages is comprised of one or more servers programmed for maintaining associations between the stored Web pages and the stored data, and receiving the identity of modified data from the database management system. In addition, the servers are capable of determining, from the identified modified data and the maintained associations, which stored Web pages are associated with the identified modified data. Furthermore, the servers are capable of communicating an update command to the cache that contains the stored Web pages associated with the identified modified data, for the purpose of updating the stored Web pages.

These and other objects, features, and advantages of embodiments of the invention will be apparent to those skilled in the art from the following detailed description of embodiments of the invention, when read with the drawings and appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of a conventional content delivery path between an end user and a Web site.

FIG. 2 is a block diagram illustrating an overview of a typical Web page delivery mechanism for Web sites with backend systems.

FIG. 3 is a block diagram showing the connection of FIGS. 3A and 3B.

FIGS. 3A and 3B illustrate an intelligent Web page caching and refreshing mechanism according to an embodiment of the present invention.

FIG. 4 is a block diagram illustrating an intelligent Web page caching and refreshing mechanism wherein an application server maintains the associations between URLs and related queries or operations according to an alternative embodiment of the present invention.

FIG. 4A shows the relationship between FIGS. 4A-1 and 4A-2.

FIGS. 4A-1 and 4A-2 show a flowchart containing source code for the creation of a URL/relevant operation mapping table by the application server in the embodiment of FIG. 4 according to an alternative embodiment of the present invention.

FIG. 5 is a block diagram illustrating an intelligent Web page caching and refreshing mechanism wherein a CachePortal.TM. API handler maintains the associations between URLs and related queries or operations according to an alternative embodiment of the present invention.

FIG. 5a is a flowchart containing source code for the creation of a URL/relevant operation mapping table by the by the CachePortal.TM. API handler in the embodiment of FIG. 5 according to an alternative embodiment of the present invention.

FIG. 6 is a block diagram showing the arrangement of FIGS. 6-1 and 6-2.

FIGS. 6-1 and 6-2 are a block diagram illustrating an intelligent Web page caching and refreshing mechanism wherein an application server converts a user request into stored procedures according to an alternative embodiment of the present invention.

FIG. 6a is a flowchart containing source code for the creation of a URL/relevant operation mapping table by the by CachePortal.TM. in the embodiment of FIG. 6 according to an alternative embodiment of the present invention.

FIG. 7 is a block diagram illustrating an intelligent Web page caching and refreshing mechanism wherein an application server proxy maintains the associations between URLs and related queries or operations according to an alternative embodiment of the present invention.

FIG. 8 is a block diagram illustrating a content change monitoring mechanism utilizing view definitions according to an embodiment of the present invention.

FIG. 9 is a block diagram illustrating a content change monitoring mechanism utilizing trigger definitions according to an alternative embodiment of the present invention.

FIG. 10 is a block diagram illustrating a content change monitoring mechanism within an external system according to an alternative embodiment of the present invention.

FIG. 11 is a block diagram illustrating the overall architecture of a dynamic content caching, invalidation, and refreshing system according to a preferred embodiment of the present invention.

FIG. 12 is a block diagram illustrating the architecture of an intelligent invalidator/refresher according to a preferred embodiment of the present invention.

FIG. 13 illustrates a timeline and an example sequence of database update events, update log retrievals, and query instances recorded by a sniffer in a dynamic content caching, invalidation, and refreshing system according to an embodiment of the present invention.

FIG. 14 is a flowchart illustrating a positive query lattice according to an embodiment of the present invention.

FIG. 15 is a flowchart illustrating a negative query lattice according to an embodiment of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In the following description of preferred embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the preferred embodiments of the present invention.

Web performance is a key point of differentiation among content providers. Crashes and slowdowns within major Web sites demonstrate the difficulties companies face in trying to deal with high Web traffic. As Internet backbone technologies have developed, many innovations in the area of service management have improved bandwidth and Web content retrieval response time. These improvements to infrastructure, however, cannot solve traffic problems at all points within the Internet. Gateway bottlenecks may result in the access time for one page of data being on the order of 10 seconds or more. Because of the gateway bottlenecks, and because there are many uncertainties along the Internet paths from end user to Web sites, content delivery networks or systems are now being developed.

Fundamentally, content delivery systems are designed for two major purposes; one is to achieve load balancing, and the other is to reduce response and access time. Embodiments of the present invention described herein reduce response and access time through content delivery systems that use intelligent caching and proxy servers to redirect users to available servers that contain the desired content and are close to or easily accessible by the users. With some of the traffic redirected, traffic surges will decrease and end users benefit from faster response time.

Embodiments of the present invention include methods and systems for intelligently refreshing dynamically generated Web content in the cache as well as in the Web server. These methods and system architectures can also be applied to static content refresh to ensure all contents in the cache are consistent with the contents in the applications and the DBMSs.

Intelligent Caching and Refreshing of Web Content

Intelligent Caching and Refreshing Mechanism 1

FIG. 3 and FIGS. 3A and 3B illustrate an intelligent Web page caching and refreshi