Digital data processor with maintenance and diagnostic system

Abstract

A state machine in a digital data processor in a UNIX-type operating system environment has state managers associated with the functional units of the data processor for indicating the state of the units; a message handler for, alternately, (a) generating requests for processing event messages indicative of conditions in the processor, that are awaiting processing and (b) generating a request for evaluation of one state manager's maintenance state, limited to transition from one state to another; and a scheduling means responsive to requests from the message handler for selectively processing the event messages, to the passage of time, and to changes of state of state managers, and for scheduling evaluation of a state manager's maintenance state. The scheduling means schedules evaluation of respective state manager's maintenance states according to a priority determined by (a) dependencies between state managers, wherein one state manager is dependent on another state manager, and (b) priorities set by scheduling conditions registered by state managers. The scheduling means executes the steps of (a) evaluating a predetermined input condition, (b) selectively making a state transition in accord with that input condition and the state manager's maintenance state, and (c) selectively performing a predetermined action associated with said transition.

Claims

What is claimed is:

1. In a digital data processor of the type having processing means for processing data including one or more functional units operatively interconnected along a bus, said functional units including any of processing units, memory assemblies, peripheral devices, peripheral device controllers and the like,

the improvement wherein said processing means includes state machine means for providing a maintenance state model of said digital data processor, said state machine means including a plurality of state managers each associated with one of said functional units for indicating the state thereof,

said processing means including event signalling means coupled to said state machine means for generating event messages indicative of conditions of said digital data processor,

at least a first said state manager being selectively responsive to any of said event messages and a state indicated by another state manager to change the state indicated by said first state manager.

2. In a digital data processor according to claim 1 the improvement wherein said processing means includes scheduling means responsive to a request for at least one of (a) processing said event messages and (b) scheduling evaluation of a state manager's maintenance state.

3. In a digital data processor according to claim 2, the improvement wherein said scheduling means includes means for evaluating a state manager's maintenance state by executing the steps of (a) evaluating a predetermined input condition, (b) selectively making a state transition in accord with that input condition and with the state manager's maintenance state, and (c) selectively performing a predetermined action associated with said transition.

4. In a digital data processor according to claim 3 the improvement wherein processing means includes message handling means coupled to said event signalling means and to said state machine means for, alternately, (a) generating requests for processing of all pending event messages and (b) generating a request for evaluation of one state manager's maintenance state.

5. In a digital data processor according to claim 4 the improvement wherein said message handling means includes means for, repeatedly and alternately, executing the steps of (a) generating requests for processing of all pending event messages and (b) generating a request for evaluation of at most one state transition of one state manager's maintenance state.

6. In a digital data processor according to claim 4 the improvement wherein said scheduling means is arranged to schedule evaluation of respective state managers' maintenance states according to a priority determined by (a) dependencies between state managers, wherein one state manager can mark itself dependent on another state manager, and (b) priorities set by scheduling conditions registered by state managers.

7. In a digital data processor according to claim 1 the improvement wherein said state managers are configured to represent a configuration of said digital data processor.

8. In a digital data processor according to claim 1, the improvement wherein said processing means operates under a UNIX-type operating system.

9. In a digital data processor of the type having processing means for processing data, including one or more functional units operatively interconnected along a bus, said functional units including any of processing units, memory assemblies, peripheral devices, peripheral device controllers and the like, the improvement wherein said processing means includes event signalling means for generating event messages indicative of conditions of said digital data processor, and state machine means for providing a maintenance state model of said digital data processor, said state machine means including

A. a plurality of state managers each associated with one of said functional units for indicating the state thereof,

B. said state managers being configured to represent a configuration of said digital data processor,

C. message handling means for, alternately, (a) generating requests for processing of all pending event messages awaiting processing, and (b) generating a request for evaluation of one state manager's maintenance state,

D. scheduling means responsive to said requests for selectively processing said event messages and for scheduling evaluation of a state manager's maintenance state, said scheduling means arranged to schedule evaluation of respective state managers' maintenance states according to a priority determined by (a) dependencies between state managers, wherein one state manager can mark itself dependent on another state manager, and (b) priorities set by scheduling conditions registered by state managers.

10. In a digital data processor according to claim 9, the further improvement wherein said scheduling means includes means for executing the steps of (a) evaluating a predetermined input condition, (b) selectively making a state transition in accord with that input condition and the state manager's maintenance state, and (c) selectively performing a predetermined action associated with said transition.

Description

BACKGROUND OF THE INVENTION

The invention relates to a maintenance and diagnostic system for a digital data processor, and particularly to such a system for a digital data processor of the fault-tolerant type.

Faults are inevitable in digital computer systems due to such things as the complexity of the circuits, and the associated electromechanical devices. To permit system operation even after the occurrence of a fault, the art has developed a number of fault-tolerant designs.

Improved fault-tolerant digital data processing systems, available from Stratus Computer, Inc. of Marlboro, Mass., include redundant functional units, e.g. duplicate CPU's, memories, and peripheral controllers interconnected along a common system bus. Each of a pair of functional units responds identically to input received from the bus. In the outputs of a pair of functional units do not agree, that unit is taken off-line, and another pair of functional units (a "spare") continues to function in its place.

Stratus Computer, Inc. has recently developed fault-tolerant systems that use UNIX-type operating systems, despite characteristics of UNIX systems that make them difficult to adapt to Stratus' fault-tolerant operation. The parent application of this one describes a Stratus system for overcoming the limitations of UNIX-type operating systems in the transfer of information to disk storage devices.

An important element of fault-tolerant systems is a maintenance and diagnostic system that automatically monitors the condition (or "state") of functional units of the data processing system, particularly those that are more readily replaceable ("field replaceable units" or FRU's). The complexity of fault-tolerant systems requires that such maintenance and diagnostic systems (or "state machines") have capabilities far beyond anything conventionally available.

It is accordingly an object of this invention to provide a maintenance and diagnostic system for a fault-tolerant digital data processing system that is efficient, reliable and fast. It is another object to provide such a system that will operate well in a UNIX-type operating system environment.

SUMMARY OF THE INVENTION

The invention provides, in a digital data processor of the type having processing means for processing data, including functional units operatively interconnected along a bus, such as processing units, memory assemblies, peripheral devices, peripheral device controllers and the like, a processor having a state machine for providing a maintenance state model of the digital data processor, the state machine including a plurality of state managers each associated with one of the functional units for indicating the state of those units. The processor includes event signallers coupled to the state machine for generating event messages indicative of conditions of the digital data processor, and at least one state manager is selectively responsive to the event messages, and to a state indicated by another state manager, to change the state indicated by the state manager.

In preferred embodiments the processor includes a scheduler responsive to a request for processing said event messages or for scheduling evaluation of a state manager's maintenance state. The scheduler includes means for evaluating a state manager's maintenance state by executing the steps of (a) evaluating a predetermined input condition, (b) selectively making a state transition in accord with that input condition and with the state manager's maintenance state, and (c) selectively performing a predetermined action associated with said transition. There may also be included a message handler coupled to the event signallers and to the state machine for, alternately, (a) generating requests for processing of all pending event messages and (b) generating a request for evaluation of one state manager's maintenance state, limited to transition from one state to another. Further, the scheduler is arranged to schedule evaluation of respective state manager' maintenance states according to a priority determined by (a) dependencies between state managers, wherein one state manager can mark itself dependent on another state manager, and (b) priorities set by scheduling conditions registered by state managers. Also, preferably, the state managers are configured to represent a configuration of the digital data processor, and the digital data processor may be one operating under a UNIX-type operating system.

DESCRIPTION OF THE DRAWINGS

Other objects, features and advantages of the invention will be apparent from the following description of a preferred embodiment of the invention, including the drawings thereof, of which:

FIGS. 1A and 1B are a schematic functional representation fault-tolerant data Processor;

FIG. 1C is a schematic illustration of a digital data processing system;

FIG. 2 is a schematic representation of state managers showing how their states may depend on each other;

FIG. 3 is a schematic representation of the architecture of the state managers and associated processes;

FIG. 4 is a formal description of an illustrative state manager specification, referred to in Appendix B (which can be found in the application file);

FIG. 5 is a flow diagram of the state transitions of a Strataslot state manager, referred to in Appendix C (which can be found in the application file);

FIG. 6 is a flow diagram of the state transitions of a virtual disk state manager, referred to in Appendix C (which can be found in the application file);

FIG. 7 is a flow diagram of the state transitions of an I/O processor state manager, referred to in D (which can be found in the application file); and

FIG. 8 is a flow diagram of the state transitions of a Stratabus state manager, referred to in Appendix D (which can be found in the application file).

BRIEF DESCRIPTION OF A PREFERRED EMBODIMENT

FIG. 1C is a schematic illustration of a digital data processing system 10 of the type used in a preferred practice of the invention. The system 10 includes a processing section 12a including central processing unit 16a, random access memory unit 18a, and partnered peripheral device controllers 20a, 21a, hereinafter IOP's. Other conventional devices are also connected to the processing section 12a of the illustrated digital data processing system. A pair of disk adapters 26a, 27a and their respective disk drives 23a, 24a, are connected with the system 10 via IOP's 20a, 21a along bus structure 28a, as shown.

In the illustrated embodiment, a single common bus structure 22a interconnects the illustrated central processing, memory and controller units to carry information between them. As discussed below, the bus structure 22a preferably includes two identical bus conductor sets, termed an A bus and a B bus, respectively, which transmit the identical information signals synchronously and simultaneously between the modules of the system 10. A third, non-illustrated bus, referred to as the X bus, carries power, timing, status and fault-notification signals.

The above-described hardware components can be selected from the variety of commercially available products and configured in a manner suitable to operate a UNIX-type operating system. Preferably, the components are selected and configured in accord with the teachings of the United States patent assigned to the assignee of the present application.

Thus, for example, in accord with the teachings of U.S. Pat. No. 4,654,857, each functional unit of the system 10 has a back-up redundant partner unit. Accordingly, processing system 10 has a second central processing unit 17a, a second memory unit 19a, and a second disk control unit 21a. Each of these units, like their respective partners, is connected to the conductor sets of the bus structure 22a. This enables all units to transfer signals to one another on either the A bus and/or the B bus, as well as on the X bus.

The basic operation of the illustrated system 10 is that, in the absence of fault, the partner central processing units 16a, 17a and partner memory units 18a, 19a operate in lock-step synchronism with one another. Thus, both units drive the A bus and the B bus identically, and both are driven identically by the two buses.

The IOP's 20a, 21a do not operate in full synchronism with one another because the disk drives 26a, 27a with which they operate are not driven synchronously. During fault-free operation, each IOP 20a, 21a writes data received from the bus structure 22a to the bus 28a so that it can be stored on the disk drives 23a, 24a.

U.S. Pat. No. 4,926,315 discloses a still more preferred construction and protocol for handling communications between the processing section 12a and the disk drives 26a, 27a with which they communicate. In accord with the teachings of that patent, partnered peripheral control units transfer control information and timing signals over a dual redundant peripheral bus to partnered interface and adaptor boards, which are connected with peripheral devices, such as disk drives 23a, 24a.

The illustrated system 10 and, more particularly, processing section 12a employs an operating system that is UNIX compatible and, more particularly, one that is compatible with UNIX System V, Release 3.2 with Berkeley Software Distribution ("BSD") extensions. Such a system is referred to by Stratus Computer, Inc. as an FTX system. Those skilled in the art will appreciate that the teachings below apply, not only to the aforementioned UNIX operating system, but to a variety of UNIX-compatible operating systems. Accordingly, the operating system 14a is hereinafter referred to as a UNIX-type operating system.

As shown in FIG. 1, that operating system includes portions 14' and 14". Portion 14' represents that part of the operating system executing within the central processors 16a, 17a for controlling their operation. The other portion 14" represents that part presently resident in memories 18a, 19a, which is available for execution by the processors 16a, 17a in lieu of, or in conjunction with, portion 14'. A further portion of the operating system may be resident on secondary storage media, e.g., disk drives 23a, 24a. The exchange of information between these portions is handled in a manner conventional in the art.

FIGS. 1A and 1B are a schematic structural representation of the fault-tolerant digital data processor 10 shown in FIG. 1C. The representation depicts functional units within the data processor 10 operatively connected along a bus. The representation shows a main cabinet 12, with slots 14 for a backpanel 16, two bulk power drawers 18, and a main cabinet fan 20. The backpanel representation 16 includes a main bus 22 (or "Stratabus"), two power supplies 24, and a clock 26. The main bus, or Stratabus, includes buses A 28 and B 30, and, functionally, a "bus" 32 which is used to represent faults that cannot be associated to either bus A or B. The main bus representation shows slots 34 (or "Strataslots") into which can be inserted CPU's 36, memory 38, or I/O processors 40.

The PK chassis 42 (see Fig 1B) is represented with slots 44 for I/O adaptors 46, and terminator cards 48. It also contains a pair of busses 50, a fan 52, and a power monitor 54. The I/O adaptor slots 44 accept I/O adapters for Ethernet, disk drawers 56, cartridge tape drives 58 and communications ports 60.

The states of these functional units are an important store of information for maintaining and diagnosing the data processor system 10. Determining those states in response to events occurring in the system, in response to the passage of time, and in response to each other's states, in a reliable and efficient manner is the goal of the invention.

The data processor includes a representation of the functional units of the processor shown in FIGS. 1A and 1B in the form of state managers arranged in a state machine For the elements of the processor 10 shown in Figs 1A and IB, the state machine of the invention provides a prime state manager which in turn creates the appropriate number of slot state managers, etc. For a data processor having a different physical configuration, of course, the state machine would create a different configuration of state managers conforming to that different data processor.

In addition, the state managers interact with each other in at least three ways, as illustrated in FIG. 2. One state manager (SM1) 62a can look to the state of another state manager (SM2) 62b as an input condition in changing its own state. One state manager (SM1) can require that if another state manager (SM2) changes its state, it should be marked "stale," that is, require evaluation. Finally, one state manager (SM1) can specify that if it and another state manager are marked stale, the other state manager (SM2) should be evaluated first.

FIG. 3 is a schematic representation of the architecture of the state managers 62 and associated processes. The MD daemon 64 is a user-level process that is event driven and that is responsible for managing the maintenance state of the data processing system 10.

Events are manifested as messages placed on a message queue 66 by the red light interrupt handler 68, the powerfail system 70, the I/O subsystem maintenance channel 72, and the error logging subsystem 74 (all parts of the operating system kernel 76) and by administrative support utilities 78. Other messages can come from the MD daemon itself, such as messages that a child process 80 created by one of the state managers has terminated or an alarm signal 82 from the operating system.

As shown in FIG. 3, the MD daemon has main portion 84 that can call on a message handler 86. The message handler 86 is the main loop of MD daemon. It dequeues event messages, calls a scheduler 88 to process those messages, and calls the scheduler 88 to evaluate state managers 62. The message handler 86 calls the scheduler 88 to evaluate a state manager 62 only when all event messages have been processed by the scheduler 88. This ensures that all possible scheduling has been done before a state manager 62 is chosen. Thus, all available event messages are processed between state manager evaluations. This ensures that the latency of MD (its ability to respond to new conditions) is no greater than the time it takes to evaluate one state manager 62. In fact the evaluation of a state manager 62 is limited to a transition from one state to another, before a check for further event messages is made.

The scheduler 88 processes event messages for the message handler 86, registers scheduling conditions for state managers 62, and chooses and evaluates state managers 62 for the message handler 86. The scheduler 86 chooses a state manager 62 for evaluation based on dependency links (the links between states of state manager illustrated in FIG. 2) and upon the priority of scheduling conditions registered by state managers 62. The scheduler maintains a running topological sort of the dependency links. This helps the scheduler 86 determine which stale state managers do not depend upon any other state managers also stale. Of those, the first with the highest priority is chosen.

A state manager 62 can create a child process 88 managed by a child manager 90 to execute time-consuming or indefinite actions, including using scripts and utilities 92. The state managers also maintain an internal representation of the configuration 94 of the data processing system 10.

The state managers use the sysftx (MD, . . . ) vendor specific system call 96 to access services 97 provided by the MD kernel. They may also access special device nodes 98 for devices 100 such as disks and communications ports.

The MD kernel calls upon utilities that can be divided into seven areas. One is the LCD manager 102 that provides control over a front panel Liquid Crystal Display unit and over a front panel red light. Others are the I/O subsystem 104, CPU management 106, memory management 108, hardware control 110, error logging 112, and device drivers 100.

A state manager 62 is the implementation of a finite state machine. It has a set of state values, and a set of transition arcs between those states. A transition arc has a boolean input expression, and a list of actions The state transition rules define transition arcs between states, and consist of six parts: (1) a rule label, (2) a state name, (3) an input expression, (4) actions, (5) scheduling condition actions, and (6) next state. The input expression can be any C language expression. This allows an input expression to invoke diagnostic tests, read hardware register values, look for and examine event messages, and even reference the state values of other state managers When a state manager is evaluated, the transition chosen is the first one out of the current state whose input expression evaluates to a non-zero value. It is possible for no transition to be chosen. The transition actions are specified by any sequence of C language statements. This allows the actions to send log messages, initiate child processes, remove event messages, perform hardware control operations, and make kernel calls.

A more detailed description of a particular embodiment of the invention in a specific data processor environment is set out in the following detailed description of a maintenance and diagnostics architecture, followed by a detailed description of a particular data processing system served by such an architecture. The details are meant to be illustrative and not limiting of the invention.

FTX Maintenance and Diagnostics Architecture

1. Introduction

This portion describes the Stratus MD software module. The description is done in 3 levels.

The first level presents a high-level, OS-independent architecture suitable for either VOS or FTX.

The second level refines this architecture into a design tailored for FTX.

The third level details this design.

The MD software module provides the software support for the fault-tolerant capabilities of the Stratus hardware architecture. It also supports non-hardware fault tolerant capabilities such as virtual disks (disk mirroring).

2. OS-Independent Architecture

The MD software module is depicted in FIG. 3. It has 4 components:

the MD daemon process (henceforth referred to as md.sub.-- daemon);

the MD event queue (henceforth referred to as md.sub.-- queue);

the MD kernel services (henceforth referred to as md.sub.-- kernel).

the MD administrative support utilities.

2.1 The Daemon Process Md daemon

Md.sub.-- daemon is a user-level daemon process. It is event-driven. It is responsible for managing the maintenance state of the Module. Module refers to the main cabinet, its associated expansion cabinets, and all the software inside. Software module refers to some component of software, such as the OS kernel, the I/O subsystem or MD).

Md.sub.-- daemon contains a dynamically evolving model of the Module. This model represents the expected configuration (Expected configuration is what MD expects to be plugged in. Deviations (such as something missing, or something plugged in where nothing is expected) will be flagged.) of the components within the module (what's plugged into what slot, what's attached to what port, what logical entities (such as virtual disks) are present), and the maintenance state of each of those components. This model is then collectively the maintenance state of the Module as a whole.

The model is dynamic because of events. Md daemon responds to 3 kinds of events:

administrative requests;

insertions and removals of hardware components;

hardware failures and errors (including power failure).

Md.sub.-- daemon responds to an event in 3 steps:

1. validate the changes in input conditions the event is alerting md daemon to;

2. make the appropriate transition in the Module maintenance state;

3. perform the actions associated with that transition.

The first step is validation. For administrative requests, validation is not necessary. The request has arrived and must be responded to. Other events are clues to md.sub.-- daemon that some hardware condition may have changed. Md daemon examines those hardware conditions to determine their actual values. A few examples of hardware conditions are:

strataboard status;

IOA status;

stratabus status;

bulk power status.

The second step is a state transition. The current Module maintenance state, together with all the input conditions, determine the next Module maintenance state.

The third step is to perform the set of actions associated with that state transition. Actions are divided into 4 categories:

relying to administrative requests;

making kernel requests;

performing hardware control operations;

issuing log messages.

One purpose of these actions is to reflect pertinent aspects of the new Module maintenance state to the OS kernel, the hardware, and the outside world.

2.2 The MD Event Queue

As mentioned above, md.sub.-- daemon is event-driven. Events are manifested as messages placed on the message queue md.sub.-- queue, by sources both inside (e.g., interrupt handlers) and outside (e.g., administrative support utilities) of the OS kernel. This queue feeds md.sub.-- daemon those messages.

2.3 MD Kernel Services

The MD kernel services (md.sub.-- kernel) are a collection of maintenance and diagnostic subroutines placed within the OS kernel. They are placed within the kernel to achieve performance, efficiency of access to OS kernel services, and efficiency of access to hardware control. These services are used by md.sub.-- daemon to do input condition validation, and to preform Module maintenance state transition actions. Examples are:

Memory test;

Disk test;

reading board status;

reading ID PROM;

access to kernel utilities to add, delete, and partner strataboards.

2.4 MD Administrative Support Utilites

Administrative requests are sent to md daemon through utility commands. These utilities place an event message on md.sub.-- queue, and wait for md daemon to reply.

With these utilities the administrator can do such operations as:

display the Module maintenance state (i.e. list.sub.-- boards);

add or delete a hardware or logical component from the Module expected configuration;

modify the MTBF threshold for a hardware component;

clear the fault history for a bus.

The MD Administrative Support Utilities will not be further described in this document.

3. FTX Implementation of the MD Architecture

FIG. 3 depicts the MD software module and its environment within an FTX Module. In the following sections, each of the components, and a brief overview of the interfaces they provide, will be described (see Section 4 and Appendix A (which can be found in the application file) for interface details).

3.1 Md daemon

Md.sub.-- daemon is a user-level process, spawned by init via inittab(4). It waits for event messages from md.sub.-- queue, using a blocking call. When one is received, md.sub.-- daemon first determines what input conditions may have changed. Not all conditions need be examined; the set of those which must be examined is a function of the current Module maintenance state as defined by the model md.sub.-- daemon maintains, and the event message received.

The current Module maintenance state and the new values of input conditions determine a maintenance state transition. This transition leads to a new Module maintenance state, and specifies a list of actions to be performed.

If the event message came from an administrative support utility, one of the actions will be to reply to the utility. This is done through a UNIX IPC message queue. The utility creates an IPC.sub.-- PRIVATE msg.sub.-- queue (see msgget(KE.sub.-- OS), and places its key in the event message prior to enqueuing the event message on md.sub.-- queue.

Md.sub.-- daemon will call md.sub.-- kernel to perform many of the state transition action (see section 3.3).

Some actions will be performed for md.sub.-- daemon by device drives on devices accessed through the appropriate special device file. Md.sub.-- daemon will use the normal mechanisms for device driver access (open, read, write ioctl(2)). However, a special open call will be used, allowing md.sub.-- daemon to bypass the normal rejection done when the driver determines the device is not available.

The device drivers will all provide ACQUIRE/RETURN ioctl(2) calls. The ACQUIRE operation allows md.sub.-- queue to ask for exclusive access to a device, in order to determine its maintenance state. The RETURN operation allows md.sub.-- queue to relinquish exclusive access, while at the same time telling the driver what the device's maintenance state turned out to be.

3.1.1 Highlights of Md.sub.-- daemon's Internal Structure

The Module maintenance state model is the primary component of md.sub.-- daemon. This model is composed of many smaller maintenance state models, one for each FRU (field replaceable unit) and each logical entity in the current configuration. These smaller models are called state manager. A state manager is a finite state automaton (i.e., state machine). Like a miniature version of the Module maintenance state model, a state manager is a model of the FRU or logical entity whose state it is managing. It responds to events in the same 3 steps:

1. validate changes in input conditions;

2. make the appropriate state transition;

3. perform actions associated with that transition.

Event messages are forwarded to the appropriate state managers, who use these as clues to possible input condition changes.

The state managers populate (are leaf nodes in) a tree data structure called the configuration representation. The configuration representation tree parallels the expected configuration of the Module. When a new component is added to the expected configuration, md.sub.-- queue creates a subtree representing that new component, and attaches it to the appropriate leaf on the current configuration representation. Conversely, when an existing component is deleted from the expected configuration, the subtree representing that component is deleted. Either of these two actions results in a new configuration representation.

Since the state managers are nodes in such subtrees, they are created and begin their work when a subtree is created. For example, when a Fresco is added to a strataslot, a subtree (these will be called fru trees in section 4) representing that Fresco will be created and attached to the strataslot in the existing configuration representation. The state manager for that Fresco, part of that subtree, begins its work by determining the current maintenance state of the Fresco, and performs the actions (like logging an insertion message if the board is physically present) called for by the first state transition taken.

The overall Module maintenance state is the composition of the maintenance state of all state managers in the configuration representation. For this composition to accurately reflect the Module maintenance state, state managers must have the ability to interact. One state manager can use the maintenance state of another state manager as an input condition. State managers can declare dependencies between one another. Such a dependency ensures that state managers are evaluated in the proper order, and ensures that when a state manager is evaluated, those state managers which depend upon its maintenance state will be evaluated as well.

State managers are not processes; they are data structures and subroutines (i.e., objects) within md.sub.-- daemon. Therefore it is important that any blocking kernel calls made by a state manager be very short-lived. Otherwise all state managers could be held up by a kernel call made from just one. This constraint is acceptable for 2 reasons.

1. A state manager can create a child process to execute time-consuming or indefinite actions; the state manager can monitor the progress of the child process as input conditions to state transitions, and can kill the child if necessary as a state transition action.

2. The IOP maintenance channel allows asynchronous operations on IOAs. The replies to these operations arrive in md.sub.-- queue as event messages, which state managers can use as input conditions. This IOP feature significantly reduces the amount of child process overhead, by eliminating the need to use child processes for IOA operations.

Script execution is another feature state managers provide. The administrator may define the contents of shell script files for FRUs and logical entities in the expected configuration. The appropriate state manager will execute the section of this script file specific to a state transition when that transition occurs. This is a limited feature however, only some transitions for some state manager types will result in script execution. This is limited for performance reasons. A typical use of script files is to download firmware into a IOP or IOA.

3.2 Md queue

Md.sub.-- queue is a queue for fixed-length messages. It allows multiple producers (at both kernel and user level) to simultaneously enqueue, and allows the consumer (md.sub.-- deamon) to dequeue in a FIFO fashion.

Md.sub.-- queue is implemented as a device driver; however, its only resource is a statically allocated buffer space in the kernel. An ioctl(2) is used to enqueue and dequeue messages. Ioctl() is chosen over read() and write() for protection; accidental re-direction of a utility's standard output into the device node for md.sub.-- queue would cause trouble if write() were allowed.

Md.sub.-- queue provides an additional ioctl(2) which blocks until 1 or more messages are placed on the queue. This is used by md.sub.-- daemon. Once this call returns, md.sub.-- daemon will remove and process all messages until none remain, at which point it will again make the blocking call.

There are 5 sources which enqueue messages:

MD administrative support utilities (user-level);

the redlight interrupt handler (kernel-level);

the Powerfail Subsystem (a kernel-level software module);

The I/O Subsystem maintenance channel (kernel-level software submodule [2]);

The Error Logging Subsystem (kernel-level software module).

As mentioned in the previous section, utilities place messages in md.sub.-- queue. The ioctl(2) call then blocks until md.sub.-- daemon has copied the utility reply back to the utility.

The redlight interrupt handler places a message in md.sub.-- queue each time a redlight interrupt occurs.

The Powerfail Subsystem enqueues two messages: one when a powerfail has been detected, the other when a powerfail ridethrough or recovery has been successful.

The I/O Subsystem's maintenance channel submodule enqueues messages containing unsolicited commands from IOPs, and maintenance channel replies to md.sub.-- daemon's asynchronous maintenance channel requests.

Some of the Error Logging Subsystem's log calls will be directed into md.sub.-- queue. These will come primarily from the DAL, VDL, and device drivers. These kernel-level software submodules will log all abnormal events which they detect in the course of normal I/O activities. These events can be used to warn MD that a change in input conditions may have taken place.

Md.sub.-- daemon can enqueue messages as well. It uses this ability at two times: whenever one of its child processes terminates, and when ever an alarm signal is received from the OS. The first use allows state managers to synchronize with the child processes they spawn. The second is used by md daemon to provide a time-out facility for state managers.

3.3 Md kernel

Md.sub.-- kernel is a collection of subroutines linked into the kernel. Md.sub.-- daemon accesses these through the vendor-specific UNIX system call. For FTX this is systftx(2). The first parameter of this system call will be "MD," causing it to call md kernel.

Md.sub.-- kernel contains major subroutines to perform sometimes complex input condition validations and state transition actions, such as ID PROM reading and diagnostic tests. It also contains minor "pass-through" subroutines which access kernel utilities.

The kernel utilities that md.sub.-- kernel uses for both its major and minor subroutines can be divided into 7 kernel areas:

LCD Manager;

I/O Subsystem;

CPU Management;

Memory Management;

kernel hardware control;

Error Logging;

Device Drivers.

3.3.1 LCD Manager

The LCD manager provides control over the front panel Liquid Crystal Display unit, and over the Module front panel red light. State managers use this to log FRU failures, and to turn on the Module red light whenever there are 1 or more FRU failures.

3.3.2 I/O Subsystem

State managers talk with the I/O subsystem through md.sub.-- kernel to converse with IOAs over the IOP maintenance channels, to invoke diagnostic tests, and to tell the I/O Subsystem of changes in the IOP maintenance state and configuration.

3.3.3 Kernel CPU and Memory Management

State managers use md.sub.-- kernel to tell these portions of the kernel about changes in CPU and Memory maintenance state and configuration. The Memory diagnostic tests also call upon kernel memory management to allocate physical and virtual memory for Memory diagnostic tests.

3.3.4 Kernel Hardware Control

Md.sub.-- kernel accesses strataboard I/O for both its major and minor subroutines, in order to reset boards, read status registers, and write control registers.

3.3.5 Kernel Error Logging

MD places log messages in the kernel error log whenever significant maintenance state changes occur. These are especially useful when tracking down the history of an elusive or intermittent error or fault.

3.3.6 Device Drivers

Some driver operations are not accessible through ioctl(2) calls, so must be made from kernel-level by md.sub.-- kernel. However, these calls still require a file descriptor, from an open call made in user-mode.

For physical disks, these operations facilitate validation of input conditions (is the disk present, is it spinning, is it formatted), and are used in the disk diagnostic tests.

4. MD Detailed Design

The purpose of this section is to describe the interfaces between MD and other FTX software modules. This will include explanations of why these interfaces are needed and how they will be used. In addition, the internal structure of md.sub.-- daemon will be presented.

The specific parameters, return codes, and associated header files for each interface subroutine will be compactly described in Appendix A (which can be found in the application file).

This section contains 3 subsections, for md.sub.-- daemon, md.sub.-- queue, and md.sub.-- kernel. Each of these will describe that component of MD in greater detail, describe the interfaces it provides to other FTX software modules, and the interfaces it is expecting from other FTX software modules.

4.1 Md daemon

FIG. 3 shows the internal structure of md daemon. There are the following components to describe:

main

message handler

scheduler

state managers

child managers and child processes

configuration representation

child and alarm signal catchers.

4.1.1 Md daemon Startup and Main

Md.sub.-- daemon is created at boot time by the init process. It communicates with init. This allows init to wait for md.sub.-- daemon to determine the maintenance state of the components of the configuration referenced in inittab (e.g. ttys and physical disks), and give them to the I/O Subsystem and device drivers.

This will be implemented with 2 lines in inittab.

md1::boot:md.sub.-- daemon:#initial invocation

md2::bootwait:mdready:#wait for md.sub.-- daemon to signal completion

The first line is placed before other boot or bootwait lines. It invokes md, which then determines the module configuration and gives the devices in that configuration to the I/O subsystem.

The second line invokes a process whose sole purpose is to wait for md.sub.-- daemon to complete the aforementioned work. It terminates when md.sub.-- daemon tells it to.

4.1.1.1 External Interfaces

The utility hw.sub.-- in.sub.-- use provides md.sub.-- daemon's first contact with the outside world. It sends a list of hardware records to md.sub.-- daemon's standard input. Each hardware record is a 2-tuple, consisting of a hardware model number and a hardware device address.

At boot time the hw.sub.-- in.sub.-- use utility gets this information from the kernel, which in turn received the information from kernel initialization, which pulled the information from the boot PROM MCT.

4.1.2 Message Handler

The message handler is the main loop of md daemon. It is called by md.sub.-- daemon's main() entry point. The message handler's job is to dequeue event messages from md.sub.-- queue, call the scheduler to process those messages, and to call the scheduler to evaluate state managers.

The following pseudocode describes the message handler's operation.

    ______________________________________
    while (TRUE){
    while (event message available OR any stale. The
    term stale, when applied to state managers, means the
    state manager's maintenance state needs to be
    re-evaluated. state managers){
    if (event message available){
              get event message
              tell scheduler to process event message
            }else{
              tell scheduler to evaluate 1 state manager
            }
    block until event message available
    }
    ______________________________________

• Inventors
  Bullis, Charles A.;
• Assignee
  Stratus Computer, Inc. (MA)
• Published
  Jun-15-1993
• Current US Classes:
  718/102
  718/103
  718/106
  719/313
  719/318
• Application #
  723065
• International Classes
  G06F 011/00
• Field of Search
  364/DIG. 395/650 371/16.5 371/48
• Examiner
  Heckler; Thomas M.
• Agent
  Lahive & Cockfield
• US Patent References:
  4486826
  4648031
  4654857
  4816990
  4866604
  4905181
  4920540
  4926315
  5020024
  5095421