Centralized diagnostic logging service

Abstract

A logging service logs diagnostic messages from applications executing in a mainframe computing environment to a centralized log. A mainframe computing environment has a plurality of mainframes coupled to a coupling facility. The coupling facility has a data storage area called a "logstream." Each mainframe has one or more subsystems executing applications. Each application preferably contains an API block, which is populated by the application and passed to a logging service API when the application desires to store a diagnostic message in the logstream. The logging service API writes the diagnostic messages to the logstream. In one subsystem, the logging service API contacts an alert facility in response to certain diagnostic messages received from applications. The logging service API in this subsystem also uses a message queue to pass the diagnostic messages to a subsystem able to write to the logstream.

Claims

We claim:

1. A system for logging diagnostic messages in a computing environment having a plurality of coupled computer systems, comprising:

a coupling facility for coupling the plurality of computer systems;

a logstream module associated with the coupling facility for storing data generated by the plurality of coupled computer systems; and

a logging service module for executing in at least one of the coupled plurality of computer systems, for receiving data representative of a diagnostic message from an application module executing on the computer system on which the logging service module is executing, for examining fields in the data representative of the diagnostic message to determine whether known data is missing, for populating the fields missing known data, and for writing the data representative of the diagnostic message to the logstream module associated with the coupling facility.

2. The system of claim 1, wherein at least one of the plurality of computer systems has a first subsystem from which writing to the logstream module is not permitted and further comprising:

a message queue module accessible from a logging service module executing in the first subsystem for receiving the data representative of the diagnostic message from the logging service module and passing the data through a message queue to a second subsystem from which writing to the logstream module is permitted.

3. The system of claim 2, wherein the first subsystem is an online transaction processing system.

4. The system of claim 1, wherein the logging service module comprises:

an alert module for determining whether to notify an alert facility for issuing an alert responsive to the data representative of the diagnostic message.

5. The system of claim 1, wherein the logging service module further comprises:

a module for examining the data representative of the diagnostic message to determine whether the diagnostic message is valid;

wherein the logging service module writes only valid diagnostic messages to the logstream module.

6. The system of claim 1, further comprising:

a report and extract module for extracting data from the logstream module responsive to selected criteria.

7. The system of claim 1, further comprising:

a browser module for browsing real-time data in the logstream module.

8. The system of claim 1, further comprising:

a legacy logging service module for executing in ones of the coupled plurality of computer systems, for receiving data representative of a second diagnostic message from a legacy application module executing on the computer system on which the legacy logging service module is executing, and for passing the data representative of the second diagnostic message to the logging service module.

9. A method of logging messages in a computing environment, comprising the steps of:

receiving data representative of a diagnostic message in an application program interface block (API) from an application program executing on one of a plurality of computer systems in the computing environment;

determining whether the API block is missing known information;

responsive to a positive determination, populating the API block with the known information; and

writing the data representative of the diagnostic message in the API block to a centralized log in the computing environment.

10. The method of claim 9, further comprising the step of:

validating data in the API block to determine whether the diagnostic message is valid;

wherein only valid diagnostic messages are written to the centralized log.

11. The method of claim 9, further comprising the steps of:

determining whether to indicate an alert responsive to the diagnostic message; and

indicating the alert responsive to a positive determination.

12. The method of claim 9, further comprising the step of:

determining a subsystem in which the application program from which the diagnostic message was received is executing.

13. The method of claim 12, wherein responsive to a determination that the application program is executing in an online transaction processing (OLTP) subsystem, the writing step comprises the steps of:

writing the diagnostic message to a message queue; and

retrieving the diagnostic message from the message queue in a subsystem other than the OLTP subsystem and writing the diagnostic message to the centralized log.

14. The method of claim 9, wherein the receiving step comprises the steps of:

receiving the data representative of a diagnostic message from the application program at a legacy logging service; and

passing the received data representative of the diagnostic message from the legacy logging service to a second logging service, wherein the second logging service is adapted to write the diagnostic message to the centralized log.

15. A computer-readable medium comprising:

a computer program product having computer-readable code embodied therein for logging diagnostic messages in a logstream associated with a coupling facility coupling a plurality of computer systems, the computer program product comprising:

a logging service module adapted for executing in at least one of the coupled plurality of computer systems, for receiving data representative of a diagnostic message from an application module executing in the computer system on which the logging service module is adapted to execute, for examining fields in the data representative of the diagnostic message to determine whether known data is missing, for populating the fields missing known data, and for writing the data representative of the diagnostic message to the logstream module associated with the coupling facility.

16. The computer-readable medium of claim 15, wherein at least one of the plurality of computer systems has a first subsystem from which writing to the logstream is not permitted, the computer program product further comprising:

a message queue module adapted to be accessible from a logging service module executing in the first subsystem and adapted for receiving the data representative of the diagnostic message from the logging service module and passing the data through a message queue to a second subsystem from which writing to the logstream is permitted.

17. The computer-readable medium of claim 15, wherein the logging service module further comprises:

an alert module for determining whether to notify an alert facility for issuing an alert responsive to the data representative of the diagnostic message.

18. The computer-readable medium of claim 15, wherein the logging service module further comprises:

a module for examining the data representative of the diagnostic message to determine whether the diagnostic message is valid;

wherein the logging service module writes only valid diagnostic messages to the logstream.

19. The computer-readable medium of claim 15, the computer program product further comprising:

a report and extract module for extracting data from the logstream responsive to selected criteria.

20. The computer-readable medium of claim 15, the computer program product further comprising:

a browser module for browsing real-time data in the logstream.

21. The computer-readable medium of claim 15, the computer program product further comprising:

a legacy logging service module adapted for execution in at least one of the coupled plurality of computer systems, for receiving data representative of a second diagnostic message from a legacy application module executing on the computer system on which the legacy logging service module is executing, and for passing the data representative of the second diagnostic message to the logging service module.

22. The computer-readable medium of claim 15, wherein the logging service module further comprises:

a first module for examining the data representative of the diagnostic message to determine whether the diagnostic message is a duplicate message; and

a second module for selectively discarding duplicate messages.

Description

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention pertains in general to data processing utilizing mainframe computer systems and in particular to diagnostic logging in such computer systems.

2. Description of Background Art

An entity that performs large amounts of data processing, such as a corporation handling a large number of daily transactions, typically relies on mainframe computer systems to perform the data processing. Not long ago, an entity could rely on a single, or only a few mainframe computers to perform all of its data processing. Each mainframe computer typically executed two or three different processing subsystems, each of which executed one or more applications. Each subsystem maintained its own diagnostic log. When a particular application generated a diagnostic message, it was relatively easy for a technician to determine which application in which subsystem generated the message, find the log associated with that subsystem, and examine the messages in the log.

Now, however, data processing needs have increased to the point where an entity such as a corporation may require dozens of mainframe computers to meet its data processing needs. The number of subsystems and applications typically executed by each mainframe computer has also increased. When an application generates a diagnostic message, it is very difficult for a technician to locate the log for that subsystem. Moreover, it is difficult for the technician to determine if other applications in other subsystems have generated the same message. Therefore, there is a need for a way to quickly and conveniently review the diagnostic logs of multiple subsystems executing on multiple mainframe computers.

BRIEF SUMMARY OF THE INVENTION

The above need is met by a logging service that logs diagnostic messages from applications executing on the various mainframe computer subsystems to a centralized log. In an embodiment of the present invention, a mainframe computing environment has a plurality of mainframes coupled to a coupling facility. The coupling facility has a data storage area called a "logstream." A preferred embodiment of the present invention uses the logging service to store diagnostic messages in the logstream. Since the diagnostic messages are centrally stored, a person can easily review diagnostic messages from multiple applications by reviewing the data in the logstream.

Each mainframe preferably executes the OS/390 operating system, also referred to as "MVS" (for Multiple Virtual Storage). Under MVS, each computer can execute one or more subsystems. One subsystem is the online transaction processing (OLTP) subsystem. In general, the OLTP subsystem executes short transactions that need to be performed right away, such as stock trading transactions. Another subsystem is the batch subsystem, which performs transactions that take a relatively long time to complete and do not need to be performed right away. For example, accounting and backup processes that can execute over long periods of time late at night can be performed on the batch subsystem.

Zero or more applications execute within a subsystem. Each application preferably contains an application program interface (API ) block, which is a data structure populated by the application when the application generates a diagnostic message. The API block is associated with a logging service API in the subsystem. The application generates a diagnostic message by passing the populated API block to the logging service API.

The logging service API in the subsystem is configured to store diagnostic messages in the logstream 116. The logging service API in the OLTP subsystem is preferably connected to an alert facility. The alert facility signals an alert to an automated system and/or a human operator of the computing environment in response to certain diagnostic messages generated by the applications. In addition, the logging service API is preferably coupled to a message queue. The message queue is utilized because the version of the OS/390 operating system used by one embodiment of the present invention does not allow direct writes to the logstream from the OLTP subsystem. Therefore, the logging service API uses the message queue to pass the diagnostic messages to a subsystem that allows logstream access. Preferably, an application executing as a long-running started batch task drains the message queue and writes the diagnostic messages to the logstream in the coupling facility.

One embodiment of the present invention includes a report and extract utility for extracting data from the logstream based on user-defined criteria. One embodiment also includes an interactive browser utility that allows a user to browse real-time logstream data.

In one embodiment, the operation of the present invention is as follows. Upon receiving a diagnostic message in the form of a populated API block from an application, the logging service preferably validates the header information in the block. Preferably, only messages having valid header information are logged. Next, the logging service determines whether there are any missing fields in the API block that can be populated by the logging service. The logging service also determines whether the block was received from an application executing in either the OLTP or batch subsystem. If the block was received from an application in the batch subsystem, then the logging service preferably writes the diagnostic message directly to the logstream. If the logging service received the diagnostic message from an application in the OLTP subsystem, the logging service preferably writes the message into the message queue. The logging service also checks the message and notifies the alert service if necessary.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level block diagram illustrating a mainframe computing environment according to an embodiment of the present invention;

FIG. 2 is a block diagram illustrating a lower-level view of the computing environment of FIG. 1; and

FIG. 3 is a flow chart illustrating steps performed by the logging service according to a preferred embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 is a high-level block diagram illustrating a mainframe computing environment according to an embodiment of the present invention. Illustrated are four mainframe computers (sometimes referred to herein as "mainframes") 110A-D, each coupled to a coupling facility 112. Although only four mainframes 110 are illustrated, embodiments of the present invention can have any practical number of mainframes coupled to the coupling facility 112. A mainframe 110 is a large, powerful computer, in contrast to personal and mini- computers. Exemplary mainframes 110 include the Z900 computer from IBM Corp. and the SKYLINE TRINIUM computer from Hitachi, Ltd. The present invention can also be utilized with non-mainframe computer systems.

Each mainframe preferably executes the OS/390 operating system, also referred to as "MVS" (for Multiple Virtual Storage). OS/390 enables open, distributed processing and allows multiple MVS's to execute on a single mainframe. Each MVS is essentially a separate, logical computer system. Each MVS preferably executes one or more subsystems. FIG. 1 illustrates an online transaction processing (OLTP) subsystem 210 and a batch subsystem 212 executing within mainframe 110A.

The coupling facility 112 is in communication with each of the mainframes 110 and provides fault tolerant capabilities to the computing environment. In one embodiment, the coupling facility 112 is utilized as an arbitrator or tie-breaker for computations performed in parallel by multiple MVS's. Physically, the coupling facility 112 is typically implemented by a mainframe executing a specialized version of OS/390. In one embodiment, a second coupling facility (not shown) shadows the operations of the first in order to provide additional fault tolerance.

The coupling facility 112 has a data storage area called a "logstream" 116. The logstream 116 can be implemented using a combination of magnetic media, electronic circuitry, and/or other forms of computer-readable memories. However, the exact configuration of the logstream 116 is not relevant to the present invention. The logstream 116 typically acts as an information store for the coupling facility 112. For example, the logstream 116 can record logs of transaction flows in order to allow database transactions to be backed-out and to support other functions performed by the coupling facility 112.

A preferred embodiment of the present invention includes a logging service that stores diagnostic messages generated by applications executing in subsystems of the mainframes 110 in the logstream 116. Since the diagnostic messages are centrally stored, a person can easily review diagnostic messages from multiple applications by viewing the the logstream.

FIG. 2 is a block diagram illustrating a lower-level view of the computing environment of FIG. 1. FIG. 2 illustrates the two subsystems 210, 212 of FIG. 1 in communication with the logstream 116. In general, the OLTP subsystem 210 executes short transactions that need to be performed right away, such as stock trading transactions. In a preferred embodiment of the present invention, the functionality of the OLTP subsystem 210 is provided by the CICS software product available from International Business Machines Corp. The second subsystem is the batch subsystem 212. In general, the batch subsystem 212 performs transactions that take a relatively long time to complete and do not need to be performed right away. For example, accounting and backup processes that can execute over long periods of time late at night can be performed on the batch subsystem 212.

In FIG. 2, the OLTP subsystem 210 is executing two applications 214A, 214B. Those having skill in the art will recognize that the number of applications executed by a subsystem can and will vary. Moreover, the exact functions performed by the applications are not relevant to the present invention, except for the functionality related to the logging service as described herein.

An API block 216 is within each application 214. The API block 216 is a data structure populated by the application 214 when the application desires to store a diagnostic message in the logstream 116. Diagnostic messages generally describe the state of the application 214 and may indicate that an application has encountered a fault, completed a task, detected an error, or any other information that may be useful to application developers, system administrators, or anyone else responsible for maintaining the computing environment. An application 214 can generate a diagnostic message at any time.

The following table describes the structure of the API block 216 according to one embodiment of the present invention. Those of ordinary skill in the art will recognize that alternative embodiments of the present invention can vary the structure of the API block 216.

                       LOGGING SERVICE API BLOCK STRUCTURE
    Level Argument Name     Format    Description
      1   LOG-API           Structure Message Block, which includes the
     following
                                      fields:
      2   LOG-SERVICE-      Structure Details of the Logging Service
          DATA
      3   LOG-SERVICE-      PIC       The name of the Common Technical Service.
          NAME              X(16)     This is preferably a constant: `LOGGING`.
      3   LOG-MAJOR-        PIC       The Major Version Number of the logging
          VERSION           9(02)     service.
                                      This is preferably a constant. Value
     depends on
                                      version.
      3   LOG-MINOR-        PIC       The Minor Version Number of the logging
          VERSION           9(02)     service
                                      This is preferably a constant. Value
     depends on
                                      version.
      2   LOG-MODULE-       PIC       This is preferably a constant `COVLOG10`.
          NAME              X(08)
      2   LOG-RTRN-         PIC       The Proc return Code.
          CODE              9(04)     PROC-SUCCESS VALUE 0
                                      PROC-FATAL-ERROR VALUE 90
      2   LOG-RETURN-       PIC       LOG-WARNING VALUES; `0010` THRU `0999`
          DETAIL            X(04)     LOG-ERROR VALUES `1000` THRU `9999`
                                      LOG-INVALID-VERSION VALUE `1000`
                                      LOG-NO-EVENT-TYPE VALUE `1001`
                                      LOG-BAD-ALERT-IND VALUE `1002`
                                      LOG-BAD-DEBUG-IND VALUE `1003`
                                      LOG-BAD-LONG-MSG-IND VALUE `1004`
                                      LOG-BAD-LONG-MSG-LEN VALUE `1005`
                                      LOG-GENLOG-FAILED VALUE `2000`
                                      LOG-ALERT-FAILED VALUE `3000`
                                      LINK-ALERT-FAILED VALUE `3001`
                                      LOG-DEBUG-FAILED VALUE `4000`
                                      LOG-CICS-ERROR VALUE `5000`
                                      LOGLINK-FAILED VALUE `5001`
                                      NOT-AUTHORIZED VALUE `5002`
                                      LOG-LOGGING-DISABLED VALUE `9999`.
      2   LOG-MESSAGE-      Structure The `Fixed` data relating to the logged
     event
          HEADER
      3   LOG-PROD-         PIC       Client's Product Delivery Channel, if
     known.
          DELIV-CH          X(02)
      3   LOG-BUS-SVCE      PIC       The Business Service related to the
     calling
                            X(08)     module.
      3   LOG-ON-           PIC       The Business Service from which the
     caller was
          BEHALF-BUS-       X(08)     invoked
          SVCE
      3   LOG-PGM-NAME      PIC       The name of the calling module.
                            X(08)
      3   LOG-USER-ID       PIC       The current UserID that invoked the
     business
                            X(20)     service
      3   LOG-UOW-          PIC       The Unit of Work Token for end-to-end
     tracing
          TOKEN             X(32)
      3   LOG-TRACE-        Structure Sub-structure within Message-Header
          DATA                        Occurs 3 times
                                      This structure contains user-defined
     event trace
                                      variables.
      4   LOG-TRACE-        PIC       User-supplied Name/Key of the Trace
     Variable
          LABEL             X(10)
      4   LOG-TRACE-        PIC       User-supplied Trace data
          VALUE             X(20)
      2   LOG-MESSAGE-      Structure Describes the actual Event
          DATA
      3   LOG-EVENT-        PIC       An Event classification code. The user
     has the
          TYPE-CODE         X(02)     freedom to define new event
     classifications.
                                      SYSTEM-ERROR VALUE `S1`
                                      SYSTEM-WARNING VALUE `S2`
                                      SYSTEM-INFO VALUE `S3`
                                      BUSINESS-REJECT VALUE `B1`
                                      BUSINESS-WARNING VALUE `B2`
                                      BUSINESS-INFO VALUE `B3`
                                      AUDIT-REJECT VALUE `A1`
                                      AUDIT-WARNING VALUE `A2`
                                      AUDIT-INFO VALUE `A3`
                                      INFO-PUB-API-INVOKED VALUE `I1`
                                      INFO-PRIV-API-INVOKED VALUE `I2`
                                      INFO-DB-API-INVOKED VALUE `I3`
                                      INFO-MODULE-INVOKED VALUE `I4`
                                      INFO-PUBLIC-API-TERM VALUE `I5`
                                      INFO-PRIVATE-API-TERM VALUE `I6`
                                      INFO-DB-API-TERM VALUE `I7`
                                      INFO-MODULE-TERM VALUE `I8`
                                      INFO-STATEMENT VALUE `I9`.
      3   LOG-ALERT-IND     PIC       Determines if ALERT must be generated
     (Y/N)
                            X(01)
      3   LOG-ALERT-        PIC       Message ID used by Mainframe Alert
     Service
          MSG-ID            X(08)     This ID is associated with a specific
     ALERT
                                      action to be taken by the Alert facility
     when this
                                      ID is detected.
      3   LOG-DEBUG-IND     PIC       Not used.
                            X(01)
      3   LOG-SHORT-        PIC       Fixed length short message
          MSG-TX            X(60)
      3   LOG-APPL-CD       PIC       Owning application of Message ID
                            X(02)
      3   LOG-APPL-MSG-     PIC       Message ID of message
          ID                9(05)
      3   LOG-LONG-MSG-     PIC       Determines if a long message is being
     passed as a
          IND               X(01)     parameter. See LOG-MESSAGE-AREA and
                                      LOG-MESSAGE-LEN-QY.
                                      The following data is optional
      3   LOG-MESSAGE-      PIC       Defines length of variable-length message
     (1 to
          LEN-QY            9(05)     32,000)

• Inventors
  Hall, Ian G.; Clark, Bernard F.;
• Assignee
  Charles Schwab & Co., Inc. (San Francisco, CA)
• Published
  Jul-8-2003
• Current US Classes:
  370/321
  702/187
  707/200
• Application #
  782901
• International Classes
  G06F 017/30; G06F 017/40
• Field of Search
  702/182 702/183 702/184 702/185 702/187 702/188 701/29 701/33 707/200 709/220 709/250 709/227 370/321
• Examiner
  Barlow; John
• Agent
  Fenwick & West LLP
• US Patent References:
  4625308
  5737600