Communications between partitioned host processors and management processor

Abstract

An inventive protocol for communicating between a management processor and host processors allows for the cooperative management of resources among host processors within a partition and also among a set of partitions in a computer system, wherein each partition may function under an instantiation of an operating system with a group of host processors. The protocol employs a message passing system using mailbox pairs in fixed but moveable or relocatable locations within the computer system shared memory. The messages share a format having specific codes or descriptors that act as codes for coordination of message interpretation. These codes include at least a validity flag and a sequence enumerator, and in a response message of a request/response message pair, a status indicator. Additionally, routing codes and function codes and code modifiers may be provided. Specific implementation details and messages are described to enable the smooth functioning of complex multiprocessor systems.

Claims

What is claimed is:

1. For use in a computer system having at least two HOST processors and at least one MIP management processor, a communications system for supporting communications between the MIP management processor and partition-identifiable ones of said at least two HOST processors comprising:

a main memory system accessible by all of the at least one HOST processors;

a set of relocatable mailbox pairs, each said mailbox pair for establishing a communications channel between said partition-identifiable ones of said at least two HOST processors and said MIP management processor, each of said set of mailbox pairs including two mailbox memory locations within said main memory system, and arranged to form a first mailbox being accessible for writing data to it by said one of said at least two HOST processors and a second mailbox being accessible for writing data to it only by said MIP management processor, and

a protocol for communicating between the MIP management processor and said one said at least two HOST processors through said communications channel, said protocol including at least

a message pair for each communication wherein each message pair comprises a request message and a reply message, wherein both'the request and the reply messages of a message pair in any said each communication comprises code written to said first mailbox by a HOST processor when a message in said pair is to be transmitted from a HOST processor to said MIP management processor, and code to be written to said second mailbox by the MIP management processor when a message in said pair is to be transmitted from the MIP management processor to said one of said at least two HOST processors, and wherein said written code includes a validity signal indicating message validity for any instantiation, a function signal indicating a message function for any instantiation of any request message, and either data signals representing message function-mediating information or a pointer to an address in said main memory where such function-mediating information can be found.

2. A communications system as set forth in claim 1, wherein each mailbox pair is useable only for communications between the MIP management processor and said partition-identifiable ones of said at least two host processors.

3. A communications system as set forth in claim 2, and wherein the locations for an initial mailbox pair is established during a boot up routine by each host processor BIOS.

4. A communications system as set forth in claim 3, wherein each mailbox pair is relocatable to a new pair of locations within said main memory system by agreement between said MIP management processor using the communications channel of the initial mailbox pair for communications between said MIP processor and said one of said at least two host processors, and wherein said one of said at least two host processors identifies said new pair of locations and communicates that address to said MIP processor via a request message pair initiated by said one of said at least two host processors.

5. A communications system comprising as set forth in claim 1 wherein said validity data signal in a MIP processor originated message may be reset by said one of at least two host processors when said one of said at least two host processors has received a message sent by said MIP processor.

6. A communications system comprising as set forth in claim 5, wherein said resetting is accomplished in said second mailbox.

7. A communications system comprising as set forth in claim 5 wherein said validity data signal may be reset by the MIP processor in a message sent by a host processor to said MIP processor.

8. A communications system as set forth in claim 1 wherein said request message of said message pair further comprises a routing code, and wherein the message pair to said written message incorporates said routing code.

9. A communications system as set forth in claim 1 wherein said reply message further comprises a status code indicating the status of the request determined by the processor receiving the request message of the message pair to which said reply written message is a reply.

10. A communications system as set forth in claim 1 wherein said request written message further comprises a sequence number.

11. A communications system as set forth in claim 1 wherein said each communication further comprises at least a second message pair.

12. A communications system as set forth in claim 1 wherein said HOST to MIP messages available to said system include at least 4 of the list of HOST to MIP messages comprising;

(NOP, Save CMOS Image, BIOS to OS Transition, Release APs, Switch to Partition Mailbox, Load Code, Map Memory, Shadow ROM, Cluster Check-in Complete, Report Fault, Write Protect BIOS/Extended BIOS Area, Switch to APIC Mode, Enable Unsolicited Messages, Enable MIP Interrupts, Heartbeat, Heartbeat Enable, MIP I/O Redirection Table--Modify, Select Sub-POD for lowest Priority, Shutdown, Change Partitioning State, Request Unit Configuration Data & Status, Request Attributes--Shared Memory, Move Mailbox, Negotiate Mailbox Version, Save and Distribute Partition Data, Request Partition Data, Report Event to MIP for Notification, Software Check-in, Request Down Memory Ranges),

and wherein said MIP to HOST message include at least 2 MIP to HOST messages, drawn from the list of reply messages comprising:

(Become AP, Become BSP, Orderly Halt, MAP available, Request Partitioning Change, Down Memory Range Notification, Host Notification--Shared Memory, Notification of Configuration Change, Partition Data Notification, Report Fault).

13. A method for communicating between a management processor and any partition-identifiable host processor in a multi-host multiprocessor computer system comprising:

establishing a mailbox pair as memory locations in a shared main memory shared by said multi-host multiprocessor computer system, for handling message traffic between said management processor and one of said host processors, a first box of said pair for receiving messages only from said management processor and for being read by said one of said host processors, and a second box of said pair for receiving messages from said one to said host processors and for being read by said management processor,

before writing a request message into either said first or second box, checking to see if the box is available,

writing by a request message into the box checked in the previous action, if said box is available, or checking again until it is available or until a timeout,

reading, responsive to a self-directed polling process, from the mailbox by said processor HOST or MIP that did not write said request message into the box,

acting on information in said request message by the processor that read said request message and

replying by the processor that read said request message by writing a response message into the other of said boxes.

14. The method of claim 13 further comprising the actions of:

resetting a valid bit in said request message by the processor that reads said request message.

15. The method of claim 13 wherein said act of establishing comprises establishing a unique mailbox pair in said shared memory for each host.

16. The method of claim 13 wherein said step of replying further comprises, first checking the one of said first or second boxes which will be used by the processor writing the reply message to see if the box is available.

17. The method of claim 13 wherein the request message written by the host is a request to the management processor to save an image file of the host CMOS memory information, and wherein the request message written contains an address for a buffer in said main memory where said host has written a copy of its CMOS memory, and wherein said management processor's responsive action is saving said image file of said buffer in a management processor memory.

18. The method of claim 13 wherein the request message written by the host is a request to the management processor to report unit configuration data and status, and wherein the request message written contains an address for a buffer in said main memory where said host has written a list of units being inquired about, and wherein said management processor's action is to place answers to the query for the units referred to in said buffer in a second buffer, and wherein said management reply message is written with an address for said second buffer.

19. The method of claim 13 wherein the request message written by the host is a request to the management processor to report memory unit attributes, and wherein the request message written contains an indication of which memory units are subject to this request message, and wherein said management processor's action is to place answers to the query for the units referred to in a buffer, and wherein said management reply message is written with an address for said buffer.

20. The method of claim 13 wherein the request message written by the host is a request to the management processor to negotiate mailbox communications level version, and wherein the request message written a highest level version which said host is capable of using, and wherein said management processor's action is to write in a reply message which version is accepted by said management processor and its associated software.

21. The method of claim 13 wherein said messages written by the Host operate to perform a function within the computer system, said function described by names of messages in the list of messages comprising;

(NOP, Save CMOS Image, BIOS to OS Transition, Release APs, Switch to Partition Mailbox, Load Code, Map Memory, Shadow ROM, Cluster Check-in Complete, Report Fault, Write Protect BIOS/Extended BIOS Area, Switch to APIC Mode, Enable Unsolicited Messages, Enable MIP Interrupts, Heartbeat, Heartbeat Enable, MIP I/O Redirection Table--Modify, Select Sub-POD for lowest Priority, Shutdown, Change Partitioning State, Request Unit Configuration Data & Status, Request Attributes--Shared Memory, Move Mailbox, Negotiate Mailbox Version, Save and Distribute Partition Data, Request Partition Data, Report Event to MIP for Notification, Software Check-in, Request Down Memory Ranges),

and wherein said messages written by the MIP wherein said messages written by the Host operate to perform a function within the computer system, said function described by names of messages in the list of messages comprising;

(Become AP, Become BSP, Orderly Halt, MAP available, Request Partitioning Change, Down Memory Range Notification, Host Notification--Shared Memory, Notificaton of Configuration Change, Partition Data Notification, Report Fault).

22. A messaging protocol for communications between a host processor and a management processor in a computer system wherein:

all host processors in said system share a common main memory, said protocol relying on individual communications channels between each host processor and said management processor, wherein each said channel comprises a pair of mailboxes, a first box of said pair being for one of the host processor's outbound communications messages to said management processor, and a second box of said pair being for outbound communications messages from said management processor to said one of the host processors, and wherein mailbox pair for said one of the host processors is located in a partition controlled by said one of the host processors, the messages in said protocol comprising paired messages, wherein outbound messages from a host processor are not equivalent to the outbound messages from the management processor, but wherein there are at least four forms of message pairs, each pair having a request message and a reply message, a first form being a request message having no pointer to a buffer paired with a reply message with a pointer to a buffer, a second form being a request message having a pointer to a buffer paired with a reply message having a pointer to a buffer, a third message pair being a request message having no pointer to a buffer being paired with a reply message having no pointer to a buffer, and a fourth form having a request message with a pointer to a buffer being paired with a reply message having no pointer to a buffer.

23. The protocol of claim 22 wherein said Host outbound request messages are drawn from a list comprising at least 4 of the list of messages comprising;

(NOP, Save CMOS Image, BIOS to OS Transition, Release APs, Switch to Partition Mailbox, Load Code, Map Memory, Shadow ROM, Cluster Check-in Complete, Report Fault, Write Protect BIOS/Extended BIOS Area, Switch to APIC Mode, Enable Unsolicited Messages, Enable MIP Interrupts, Heartbeat, Heartbeat Enable, MIP I/O Redirection Table--Modify, Select Sub-POD for lowest Priority, Shutdown, Change Partitioning State, Request Unit Configuration Data & Status, Request Attributes--Shared Memory, Move Mailbox, Negotiate Mailbox Version, Save and Distribute Partition Data, Request Partition Data, Report Event to MIP for Notification, Software Check-in, Request Down Memory Ranges),

and wherein said MIP outbound request messages are drawn from a list comprising at least 2 messages, drawn from the list of reply messages comprising:

(Become AP, Become BSP, Orderly Halt, MAP available, Request Partitioning Change, Down Memory Range Notification, Host Notification--Shared Memory, Notification of Configuration Change, Partition Data Notification, Report Fault).

Description

A portion of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to protocols for multiprocessor computer systems and processes therein, and has particular applicability to the Intel processors. The protocol may be used in a multiprocessor configuration with only and may be applicable to multiprocessor configurations with only non-Intel type processors as well processors or in a computer system having more than one type of processor, including non-Intel processors. (A computer system is to be thought of as any system for processing data having a single "main" memory, at least one instruction processor (a "Host" processor although in preferred configurations, a Host itself will be comprised of more than one processor) and at least one input/output (I/O) device associated therewith and controlled thereby. The invention relates to multiple processor computer systems and communications between a Host processor and a management processor that is available for managing operations of the multiprocessor computer system.)

2. Background Information

To provide flexibility in operations of multiprocessor systems using Intel processors has proven to be a challenge due to the inherent design attributes of such processors, as well as the design attributes of the operating systems which run with them. To do it in a way which allows for heterogeneity of processors and operating systems in a single computer system is even more difficult.

Described herein is a system having a protocol for communication by all the "host" processors on a computer system with a management system processor. In the preferred form, Intel brand processors or ones which follow their specifications, are used for all host processors and for the management processor(s) in the computer system. The management system processor coordinates the activities of all the host processors, accommodating the host processors' needs for system resources as communicated to the management processor.

It is important to keep in mind various valuable attributes of multiprocessor systems that allow for such a form of management system to allocate the system resources dynamically. Specifically, the protocol used should allow a management system to monitor the host system(s) and perform valuable functions to make the overall system run efficiently. Some of the functions should include partitioning into multiple partitions (each partition having an instantiation of an operating system), controlling the execution of the host operating system (start, stop, boot, etc.) and viewing the status of hardware units in the system. Ultimately, the management system should have control over the host(s).

The protocol should also be flexible, providing new messages and formats without changing the protocol itself. Also, from the Host system's perspective, the management system should look like any other I/O device to allow the management system to be connected to the host systems through any I/O port, giving substantial flexibility in how the whole system can be managed and from where. Also, it would be advantageous to be able to direct host systems as well as the management system to buffer areas in memory where substantial chunks of information can be passed.

In U.S. patent application Ser. No. 09/120,797, now abandoned entitled: COMPUTER SYSTEM AND METHOD FOR OPERATING MULTIPLE OPERATING SYSTEMS IN DIFFERENT PARTITIONS OF THE COMPUTER SYSTEM AND FOR ALLOWING THE DIFFERENT PARTITIONS TO COMMUNICATE WITH ONE ANOTHER THROUGH SHARED MEMORY, (incorporated herein in its entirety by this reference thereto) for example, it is taught how a computing system can be constructed so that a problem can be shared between partitions, leading to increases in throughput. In this cited document a number of other systems which can be employed in multiprocessor environments are mentioned in the background section. All such systems can be enhanced and take advantage of having a unique management node which can organize the manner in which the processors in a multiprocessor computer system employ the resources of the system. (Providing a redundant management node is not an unreasonable enhancement.) The management node can cooperatively reconfigure the resources of the computer system to allow for removal of parts of the system and their replacement without ever stopping the overall computer system functioning. Thus, in highly interactive transaction processing systems, for just one example, the transactions can continue to execute while segments of the computer system are removed from the system and replaced for upgrade or repair, even including sections of the main memory. Information from banks of sensors can continue to be processed while a processor or group of processors is swapped out, for another example. I/O subsystems can be brought down or added without stopping the entire system for the change, for a third example.

In U.S. patent application Ser. No. 09/362,388 now U.S. Pat. No. 6,687,818 entitled: METHOD AND APPARATUS FOR INITIATING EXECUTION OF AN APPLICATION PROCESSOR IN A CLUSTERED MULTIPROCESSOR SYSTEM, also incorporated herein by this reference in its entirety, a boot process for multiprocessor systems is described that uses some of the protocol which will be described in detail herein.

The relative ubiquity of Intel microprocessor architectures, from the so called x86 to the IA32 and IA64 and the concomitant ubiquity of Microsoft NT-type operating systems makes it important to be able to employ Intel's design in such multiprocessor systems. Enhancing commercial compatibility in a multiprocessor environment with what most of the world uses in its computing environments becomes achievable through the use of an Intel Host Protocol. With a properly configured BIOS, one can use the inventively designed protocol communications to direct the activities of such processors to memory locations that assign specific resources to each such processor. However, the difficulties are multiplied by the fact that all memory locations in a multiprocessor computer system that employs a single memory unit (or single memory unit group for the main memory function for all the processors) must be assigned to each Intel processor so that the individual memory map of the Intel processor (or whichever other processor type might employ this inventive protocol) operates effectively in such a system.

When in operation in a multiprocessor system, these processors, are called "host" processors since they provide the processing hardware services to the applications and operating system software for their partition of the computer system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a multiprocessor computer system for use with the invention.

FIG. 2 is a block diagram of a multiprocessor computer system similar to that of FIG. 1 with substantially more detail.

FIG. 3A is block diagram of a preferred embodiment linker system for communication of the protocol between at least one management processor system and the processors of the computer system.

FIG. 3B is a block diagram identifying how the linker system of FIG. 3A establishes an electrical pathway between a MIP system and an MSU.

FIG. 3C is a block diagram identifying how the linker system of FIG. 3A establishes an electrical pathway between a MIP system and an individual host processor on a Sub-POD board.

FIG. 4 is a block diagram of an integrated management system.

FIG. 5 is a simplified block diagram of the elements of a mailbox system for use with this invention.

FIGS. 6 and 7 are tables of messages for use in preferred embodiments of the protocol of this invention.

FIGS. 8 and 9 are block flow diagrams of how the Host to MIP and MIP to Host messages are passed through the mailboxes in the preferred embodiments of this invention.

SUMMARY OF THE INVENTION

An inventive protocol, its features, functions and operative elements for communicating between a management processor and host processors is described herein. This protocol allows for the cooperative management of resources among a plurality or multiplicity of host processors within a partition and also among a set of partitions in a computer system, wherein each partition may function under an instantiation of an operating system with a group of host processors.

Specifically the protocol employs a message passing system using mailbox pairs in fixed but moveable or relocatable locations within the computer system shared memory. The messages share a format having specific codes or descriptors that act as codes for coordination of message interpretation. These codes include at least a validity flag and a sequence enumerator, and in a response message of a request/response message pair, a status indicator. Additionally, routing codes and function codes and code modifiers may be provided.

The message passing system provides a basis for conversations between host and Management processors to provide a heretofore unavailable level of control over the system. These conversations or dialogs comprise a set of message pairs that can allow coordination of maintenance activities, allocation of system resources and adaptive use and modification of the protocol itself.

Short usage process algorithms are described that enable use of the mailboxes with these codes.

For the protocol to function smoothly, each partition will have its own mail box pair and communication between a multiprocessor host partition and a management processor will be coordinated by a single processor in the host, designated the "BSP" (for "Boot Strap Processor"). The management processor and its supporting hardware can be thought of as a node on the computer system. It, together with the software that provides management functions is also called an Integrated Management System or IMS.

Both the management processor (sometimes called the "MIP", running under the control of an Integrated Management Software program or "IMS") and a host BSP will use memory within the partition owned by that host for these mail box pairs. The same address translation hardware can advantageously be used to keep the system software parsimonious, but this requires hard-wiring of connections between each BSP processor module and the management processor. This hardwiring configuration is preferred but not required. In this preferred form, a linking module in a crossbar interconnect (which links the processors with the computer system main memory) is directly connected to "feedthrough" pins in a MIP node. This limiting module enables the routing of signals directly from the management processor to the BSP and other processors within the BSP's domain, by passing messages through the address translation hardware of the crossbar interconnect (which is also used by this BSP) to reach main memory locations, and through the address translation hardware of the Sub-POD on which each processor in the BSP's domain is physically located to provide appropriate address translation for the crossbar translator hardware. This linking module may in some embodiments also allow for direct passing of signals to the processor, for example, to let the processor out of a reset condition.

A less preferred solution to communicating messages in paired mailboxes in fixed and moveable locations as taught herein could employ software translation from domain to domain across the system wide main memory unit of the addressing schemes from each of the partitions "controlled" by the management processor. In such a system, the need for hard-wired connections to individual processors would be obviated.

An I/O port fixed location could also be substituted where the I/O port address translation hardware handles the conversion for each message to specified processors.

Yet another system could use dedicated real memory locations in the system main memory for mail box messaging for each processor, but some of the flexibility provided by the preferred embodiment would be lost.

A list of messages currently used in this inventive protocol and their use in a multiple processor computer system environment is detailed within. Also described is the system of dialogues available with the protocol using the messages to coordinate the control of the multiprocessor system.

To provide additional flexibility, one of the message pairs allows for negotiation of versions between the host and the management processors so that new host software entity that communicates with the management processors, including but not limited to BIOS versions can be included in the multiprocessor system and yet remain coordinated within the overall system, and so that additional messages may be developed and used in the protocol as design considerations may warrant. Another message pair allows for relocation of the message pair mailbox locations within the main memory system controlled by a host. In all, 39 message pairs are described for the preferred embodiment system, with only 10 of those being originated by the MIP.

Many other features and limitations are described in the detailed description below.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The multiprocessor system should first be described to enable an understanding of the way the protocol can function in it. Then the protocol itself can be described. Finally, the operation of the protocol in the system can be described, and conclusions drawn. Accordingly, the detailed description is in four parts.

Part 1. Preferred Embodiment System Description

FIG. 1 illustrates a multi-processor system that includes at least the processor modules 110, 112, and 114. Processor modules 110, 112 and 114 are of comparable compatibility in that they at least can communicate through the system interconnect to the main memory and I/O using the same protocol. In the preferred embodiments all of these processor modules will be Intel architecture processors, however, the present invention further contemplates that heterogeneous processors and/or operating systems can co-exist. Each processor module 110, 112 and 114 is self-contained. The processor modules 110, 112 and 114 can each include a plurality of processors. Two or more of processor modules 110, 112 and 114 share access to main (or global) memory 160 and/or to I/O devices 120, 122, and 124, typically through a system interconnection mechanism, such as system interconnection 130. In preferred embodiments this interconnection device is a non-blocking cross-bar switch. Processor modules 110, 112, and 114 can communicate with each other through main memory 160 (by messages and status information left in common data areas) as defined in some detail in the previously incorporated patent application U.S. patent application Ser. No. 09/120,797.

Also in the preferred embodiment, one or more processor modules can be configured, as a separate partition within the computer system, such that multiple partitions may exist within the computer system, each partition operating under the control of a separate operating system. For example, each processor module 110, 112 and 114 of FIG. 1 can be defined as a separate partition controlled via a separate operating system 170, 172 and 174. As employed in one preferred embodiment, each operating system 170, 172 and 174 views main memory separately as though each is the only entity accessing main memory 160, although it is possible that a single operating system could be shared among more than one of the processors 110, 112, and 114.

Also in the preferred embodiment, each of the processors illustrated in FIG. 1 is a unit containing a plurality of processors functioning together as a single computing entity within a partition. Refer to the description of FIG. 2 for detail regarding this aspect of the configuration.

A distinction should be made between multi-processor systems and multi-computer systems. A multi-computer system is a system in which computers are interconnected with each other via communication lines to form a computer network. The computers are autonomous and may or may not communicate with each other. Communication among the computers is either via fixed paths or via some message-switching mechanism. On the other hand, a conventional multi-processor system is controlled by one operating system that provides interaction between processors and all the components of the system cooperate in finding a solution to a problem. Using the invention described herein, a multiprocessing system may share many system resources, including even main memory, among more than one instance of an operating system and/or among more then one kind of operating system.

In the multiprocessor system 100 of FIG. 1, there is also an entity for managing the system called an Integrated Management System or IMS 115 which contains a management processor or MIP. This MIP processor is compatible with the inventive protocol. Since the protocol was designed to enhance management communications with the Intel IA32 and IA64 architectures, at least these chips will be compatible with the protocol.

The IMS system in the preferred embodiment contains a link to a hard-wired connection to each of the processors through a link on the circuit board on which the processor resides, by which the IMS can let these host processors out of reset, an action which allows them to boot themselves up. This hard-wired connection is illustrated as line 117. The routing of this bus can also be used for other purposes. For example, on the Intel chips currently sold and those anticipated, a pin called RESET must be addressed directly in order to cause a reset which when released will cause or allow the processor chip to run using the memory location it will look to load a boot program called a BIOS. It is possible to connect some other way to this pin from off the processor's board, but at the present time, it is preferred to employ a unique bus for this purpose.

For the instant invention, the protocol messages from the IMS MIP are passed over a hardwired connection 117 to the memory locations through the address translation modules associated with the processors linked through the connection 117. Because an electrical bus such as this is in place in our computer systems, it is advantageous to use it to communicate all messages to the processors from the IMS, and vice-versa through such a pathway. As mentioned in the Summary section above, other communications pathways for the protocol can be established using direct to processor communications pathways or through I/O bridges 116b and 116a, respectively, however, in our system it is preferred to use a linker system which here is illustrated by line 117.

Details of how a preferred system for employing this protocol can be booted up are described in previously filed U.S. patent application Ser. No. 09/362,388, now U.S. Pat. No. 6,687,818 entitled: METHOD AND APPARATUS FOR INITIATING EXECUTION OF AN APPLICATION PROCESSOR IN A CLUSTERED MULTIPROCESSOR SYSTEM, incorporated herein in its entirety by this reference.

Once the processors are running, communication between them and the IMS processor (Management Instruction Processor or MIP) is accomplished by use of the linker system 50, described with reference to FIG. 4 in detail below.

Please now refer to FIG. 2 which is an illustration of a preferred embodiment of a computer system 200 for use with the present invention. Computer system 200 includes a main memory, illustrated here as main memory 160, and a plurality of processing modules A-D (having parts surrounded in dotted lines for clarity and sometimes called "Sub-PODs") connected to the main memory via respective third level cache modules 230A-D and crossbar interconnects 290A and 290B.

In the present embodiment, main memory 160 is a directory-based memory system and is capable of supporting various memory consistency models. Main memory 160 includes a plurality of memory storage units (MSUs) 220, such as memory storage units 220A, 220B, 220C, and 220D. Preferably, each memory storage unit 220A, 220B, 220C, and 220D includes at least eight gigabytes of memory. Preferably too, each memory storage unit 220A, 220B, 220C, and 220D includes sixteen semi-independent banks that share four double-wide data busses and eight unidirectional address busses. Alternatively, we could provide a unitary MSU of any size, rather than the set of separate units, but multiple memory units provides additional flexibility to customers so such systems are currently preferred.

The plurality of third level cache modules 230, such as third level cache modules 230A through 230D, include a plurality of third level cache translators that preferably include application specific integrated circuits (called TCTs), such as TCTs 270A through 270H. In the present embodiment, pairs of processors (e.g., 240A and 240B) share a common bus (e.g., 280A) with a single TCT (e.g., 270A) within a given third level cache (TLC) in a Sub-POD (e.g., 230A). Each TCT 270 performs address relocation, reclamation, and translation for memory addresses issued by the processors to which it is connected, so that the processor has the same kind of memory to work from as its processor was designed to work with, regardless of the pagination of the main memory 160's organization. The details of how such translation can be accomplished with memory sharing are described in U.S. patent application Ser. No. 09/120,797. It is sufficient for our purposes here to mention that when one of the processors of a processing module of a given partition issues an address on its address lines ("the referenced address" or "processor address"), the TCT 270 for that processor adjusts the address for any relocation, reclamation, or shared windowing, as required, to produce the address of the corresponding location in the main memory 160. As described in the U.S. patent application Ser. No. 09/120,797, now abandoned the TCTs 270 provide a means mapping the physical address space of the processors in each partition to the respective exclusive memory windows assigned to each partition, and, more specifically, a means for relocating a reference to a location within the physical address space of the processors on a respective partition to the corresponding location within the exclusive memory window assigned to that partition. As also described, in a similar manner, the TCMs 285 perform any relocation or reclamation required for memory addresses received from an I/O processor (e.g., PCI card) communicating via a DIB 250A or 250D and TCM to main memory.

Each third level cache module 230A through 230D is connected to a respective plurality of processors (MPs) 240A through 240S. Specifically, in the present embodiment, each TLC 230 is connected to up to four processors. Each TLC 230 and its respective group of MP processors define a Sub-POD. Further according to the present embodiment, two Sub-PODs are connected via a crossbar interconnect (e.g., crossbar interconnect 290A or 290B) to form a POD. Thus, in the embodiment illustrated in FIG. 2, there are four Sub-PODs connected via crossbar interconnects 290A and 290B, respectively, to form two PODs.

In operation, memory storage units 220 bi-directionally communicate with third level cache modules 230, through crossbar interconnects 290. Crossbar interconnects 290 bi-directionally communicate with direct I/O bridges 250 via I/O buses 210, and with processors 240 through TCTs 270. Direct I/O bridges 250 bi-directionally communicate with peripheral component interconnects 260.

In the present embodiment, the processors (MPs) 240 may comprise Intel processors (e.g., Pentium Pro, Pentium II Xeon, Merced), Unisys E-mode style processors (used in Unisys A Series and ClearPath HMP NX enterprise servers), or Unisys 2200 style processors (used in Unisys 2200 and ClearPath HMP IX enterprise servers. It is contemplated that processors from other manufacturers with compatible addressing schemes and work lengths or with translations suitable to those otherwise compatible could be used. Preferably, a given Sub-POD employs four processors of the same type. However, the present invention contemplates that different Sub-PODs may employ different types of processors. For example, one Sub-POD may employ four Intel processors, while another Sub-POD may employ four Unisys E-mode style processors. In such a configuration, the Sub-POD that employs Intel processors may be defined as one partition and may run under the control of an Intel-compatible operating system, such as a version of Unix or Windows NT, while the Sub-POD that employs Unisys E-mode style processors may be defined as another partition and may run under the control of the Unisys MCP operating system. As yet another alternative, the Sub-PODs in two different partitions may both employ Intel processors, but one partition may run under the control of an Intel compatible operating system (e.g., Windows NT), while the other partition may run under the control of the Unisys MCP operating system through emulation of the Unisys A Series computer architecture on the Intel processors in that partition. Various other configurations are certainly possible and contemplated so that for example the UnixWare or Solaris operating systems as well as others could be used without exceeding the scope of the invention.

Additional details of the architecture of the preferred embodiment of the computer system 200 of FIG. 2 are provided in the following co-pending, commonly assigned applications listed following this paragraph, each of which is incorporated herein by this reference in their respective entireties.

"A Directory-Based Cache Coherency System," Ser. No. 08/965,004, now abandoned Filed Nov. 5, 1997.

"Split Lock Operation To Provide Exclusive Access To Memory During Non-Atomic Operations," Ser. No. 08/964,623, Filed Nov. 5, 1997.

"Message Flow Protocol for Avoiding Deadlocks," Ser. No. 08/964,626, now U.S. Pat. No. 6,092,156 Filed Nov. 5, 1997.

"Memory Optimization State," Ser. No. 08/964,626, now U.S. Pat. No. 6,052,760 Filed Nov. 5, 1997.

"System and Method For Providing Speculative Arbitration For Transferring Data," Ser. No. 08/964,630, now U.S. Pat. No. 6,049,845 Filed Nov. 5, 1997.

"High Performance Modular Memory System with Crossbar Connection," Ser. No. 09/001,592, now U.S. Pat. No. 6,480,927 Filed Dec. 31, 1997.

"Programmable Address Translation System," Ser. No. 09/001,390, now U.S. Pat. No. 6,356,991 Filed Dec. 31, 1997.

"High-Speed Memory Storage Unit for a Multiprocessor System Having Integrated Directory and Data Storage Subsystem," Ser. No. 09/001,588, now U.S. Pat. No. 6,415,364 Filed Dec. 31, 1997.

"Directory Based Cache Coherency System Supporting Multiple Instruction Processor and Input/Output Caches," Ser. No. 09/001,598, now U.S. Pat. No. 6,587,931 Filed Dec. 31, 1997.

"Bi-directional Interface Distributed Control Mechanism," Ser. No. 09/096,624, now U.S. Pat. No. 6,182,112 Filed Jun. 12, 1998.

"Source Synchronous Transfer Scheme," Ser. No. 09/097,287, now U.S. Pat. No. 6,199,135 Filed Jun. 12, 1998.

"Queuing Architecture for Use in a Data Processing System Having Independently-Operative Data & Address Interfaces," Ser. No. 09/096,822, now U.S. Pat. No. 6,178,466 Filed Jun. 12, 1998.

A POD is defined as a unit having four MP processors (in this embodiment), two TLC's and a crossbar interconnect module with it's two TCM's. A POD could have less than the maximum number of processors in it, if desired, and may even have only one Sub-POD with a single processor in that Sub-POD if desired.

In the present embodiment, a maximum configuration of the computer system 200 includes four PODs as described above, which would be double in size of the computer system 200. Thus, in the maximum configuration, a computer system would include (4 PODs) * (8 processors per POD)=32 processors. Computer system 200 can be partitioned on any combination of POD or Sub-POD boundaries. It is understood, however, that the present invention contemplates other multiprocessing environments and configurations. For example, computer system 200 could be expanded by plugging in more memory storage units 220 and more PODs or Sub-PODs.

PODs can be defined to include direct I/O bridges (DIBs) 250A and 250B which communicate with main memory through the crossbar of the respective pod.

Further according to the present invention, multiple partitions within the computer system, each of which may comprise one or more PODs or Sub-PODs, each operates under the control of a separate operating system. The operating systems executing on the different partitions may be the same or different. For example, the present invention contemplates an environment wherein at least two of the operating systems are different and one operating system does not control or manage the second operating system.

Refer now to FIG. 3, in which the Linker System 50 is shown comprised of at least MIP (Management Instruction Processor), a series of communication lines 117d, and tap linker buses or switching hardware 60 and 63. In this block diagram, a redundant MIP system is also shown which provides additional flexibility to the architecture. This MIP1115s has its feed-through signal lines indicated as 117c. Accordingly, explanation will refer only to the single MIP system illustrated as MIP0115r and it will be understood that this explanation applies also to MIP1.

Timing coordination of the overall system is provided through an oscillator block 71 which has its own tap linker 61. This provides essential clocking features so that all the signals are coordinated in time. In the preferred embodiment the MIP issues a maintenance enable signal on the ME lines whenever it wants to send a communication. As was illustrated in FIG. 2, the multiprocessor computer system comprises a series of Memory Storage Units or MSU's like MSU's 220q-220t. Each one of these MSU's will have a tap linker 60 for communicating messages from and to the MIP. (The addressing and control of the MSU's are controlled through ASICs 61-65 as is understood in the art, here having two types, one MCA for address and control and the others for passing data). The tap linker system is not required for the communications pathway between the processors (like IPs 240A-D), as will become clear in discussion of the FIG. 3B diagram describing the use of the linker to allow the MIP to communicate with mailboxes in the MSU (like 220Q).

As was also described with reference to the computer system, a number of PCI modules 41, 42, are connected to the crossbar interconnect 43 through a data interface bus DIB 43. Also, as was illustrated in FIG. 2, a pair of Sub-PODs 31 and 32 may be connected through a crossbar interconnect 43 to the memory storage units. To simplify the explanation, a memory storage unit functional interface 45 is illustrated here which provides a link between the linker 63 as well as the Sub-PODs (31) via the crossbar interconnect or system interconnect here illustrated as CI43. (The system of interconnects between the MSU's and CI's which is the system of ports labeled 45 in this illustration and used in this preferred embodiment is described in substantially more detail in U.S. patent application Ser. No. 09/001,592, incorporated herein in its entirety by this reference. However, it is sufficient to note for the purposes of this discussion that there is a pathway independent from the tap linkers that provides a data channel between the PODs and the MSUs.) The maintenance enable signals are used by the linker 63 to allow only one (1) MIP to communicate with the linker 63 at any time. In the preferred embodiment a maximum of two (2) MIPs may be used in a computer system.

Messages in the protocol, may be sent on the signal line 119 to the linker 63 by the MIP. The linker 63 takes advantage of the translation hardware TCM 41 and TCM 42 to map the message to the appropriate space in the memory unit wherein the mail box the MIP wishes to write to is located. The linker 63 transfers this message data across the MSU functional interface 45 to the appropriate location in the main memory. (More detail regarding this transfer is had in the description of FIG. 3B). As was illustrated in the earlier figures, the main memory can be comprised of a set of MSU units and the address translation hardware will have been configured for the appropriate partition prior to the MIP using the address translation hardware in crossbar interconnect 43.

At start up, all host processors will be pointing to the same main memory mailbox locations and the MIP will use the address translation hardware of the main memory in as unit 6.0.

It should be noted that an additional signal line 120 is provided to a linker unit 60 within a MSU 220q. This signal line 120 may be used to load messages into mailboxes prior to the boot sequence being completed.

Referring to FIG. 3B, the block diagram 400 illustrates electrical pathways by which the MIP system 401 can communicate through to the main memory storage unit system 408. The common scan interface 402 is the equivalent of the communication feed-through lines 117d in FIG. 3A. Thus, line 120a is the equivalent of line 120 in FIG. 3A. This allows for direct transfer of data into the linker system 463 in the crossbar interconnect system 406. You will recall from FIG. 2 that the crossbar interconnect system contained a pair of ASICs which are here illustrated as block 467 which supply address translation capability to the system so that the host processor using such ASIC can receive information from an address it asks for from what that address corresponds to within the MSU. Since the integrated management system software running the MIP 401 is aware of which processor it is communicating with, it is also aware of the range of memory to which this ASIC has access in the main storage unit 408. Initially, the address is a predetermined (fixed) address which is within the range of addresses which may be accessed by that processor and by the ASICs. One of the messages of the protocol allows the host Operating System to change the address at which messages are read/written. The ASIC 467 will then route the data directly to the main storage unit mailbox for the MIP across line 120b, just the way it would if the request to send information to the MSU were coming from the host processor, using the Host's address translation hardware and the Host's in addressing the Host's memory range. This same pathway can be used for reading and writing memory locations by the MIP in the main storage unit, with no intervention by the instruction processor running the partition in that main storage unit to which the MIP communication is directed. In the preferred embodiment, requests from a host processor IP 404 or a MIP, will have equal priority to employ the ASIC translation function and the ASIC block 467 will serve them on a first come first served basis.

Note that there is also connection via the feed-through or common scan interface 402 to a PCI-bridge, DIB403. This allows the MIP to communicate data through the same ASIC 467 as if the MIP is a peripheral subsystem of the overall computer system.

Finally, as is described in more detail in FIG. 3B, an additional direct electrical pathway is set-up between the MIP through the common scan interface 402 to each instruction processor 404. The instruction processor communication through the Sub-POD 405 and the crossbar interconnect 406 is illustrated in more detail in FIG. 2.

Referring now to FIG. 3C, the MIP system 451 with its common scan interface 452 is shown connected through the linker system 464 in POD 453 to the instruction processor 0 (457) in Sub-POD board 455. This electrical pathway is consistent with those indicated as t.sub.1-16 in FIG. 2.

The memory translation hardware ASICs 454a and 454b, it should be noted, are the equivalent of box 467 in FIG. 3B. These ASICS are connected through the linker subsystem 464 in POD 463, thus, the communication pathway through the linker 464 can be used to reach the partition memory locations for the mailbox addressable by the MIP.

The MIP system 115a is illustrated in more detail in FIG. 4. The signal lines and maintenance enable lines extend from the MIP hardware box 115a (also referred to generally as running the Integrated Management System or IMS software) as line 117b. This is connected to a PCI card containing the MIP end of the CSE(common scan engine) 84 which in turn is connected to a system bus 81 controlled by the CPU (or Instruction Processor) 82. A memory sub-system 83 and input/output controller 85, provide the functional elements necessary for the software to operate as an Integrated Management System (IMS), employing the inventive protocol to monitor and control the multiprocessor computer system (not shown in this figure).

Part 2. Preferred Embodiment Protocol

The protocol functions in the multiprocessor computer system to provide a process and system to communicate between the MIP and the host processors. The communications are by way of a mailbox system which is described in the next section. The format of the mailbox request message is the same for all messages used in the protocol. There are two basic forms, a REQUEST and a RESPONSE message. These in the preferred embodiment have particular forms but so long as the required elements for the functions are present various forms can be used.

The REQUEST form is organized as a 00-3F.sub.hex bit (64.sub.10 bit) message format. This is the same size as the Intel cache line. Therefore it is convenient, and therefore preferred, to use the same size messages for replies as requests, even though the replies could be made shorter since they don't really need to carry a buffer address. (Since the cache line is going to cost as much time to write as a shorter message, it is unlikely that there would be any benefit from shortening the message size). Likewise, in designing the form of the message it could be made shorter if the buffer is in a separate piece and two lines are used, however, again, sufficient space for a buffer is available in the Intel cache line and therefore with the additional simplicity and reduced cost and complexity the message format design we teach is preferred. Only one order preference should be mentioned, however. The valid flag is in the last byte in the mailbox to allow the MIP software to write the mailbox in one operation and be assured that the valid flag will be written last, independent of the hardware implementation. This will allow the mailbox message size to grow beyond both the one cache line and also if the next generation of processor uses a larger cache line, this too can easily be supported.

The form is set out in the following table:

                                          TABLE 1
                                  REQUEST mailbox format.
                                         ##STR1##

                                          TABLE 2
                                 RESPONSE mailbox format.
                                         ##STR2##
    function code: SAME EXPLANATION AS REQUEST
    MIP routing code: SAME EXPLANATION AS REQUEST copied from sender if this is
     a Host .fwdarw. MIP message.
    host routing code: SAME EXPLANATION AS REQUEST copied from sender if this
     is a MIP .fwdarw. Host message.
    code modifiers: SAME EXPLANATION AS REQUEST
    sequence number: copied from request mailbox
    buffer address: SAME EXPLANATION AS REQUEST
    buffer size: SAME EXPLANATION AS REQUEST
    status: result of the request determined by recipient
    valid flag: SAME EXPLANATION AS REQUEST

                             TABLE 3
                       Routing Code Summary
                      Description
        MIP Routing
        Code.sub.16
        0             Not used
        1             Platform Messages
        2             Architectural Messages
        3             Mailbox Handler Messages
        4             Shared Memory Control Messages
        Host Routing
        Code.sub.16
        0             Generic Message
        1             BIOS Messages
        2             HAL (Hardware Abstraction Layer)/PSM (Unisys
                      equivalent of HAL) Control Messages
        3             Processor Messages
        4             I/O Messages
        5             Exclusive Memory Messages
        6             Shared Memory Messages
        7             MIP/Host Communications Control Messages
        8             Partition Data Messages
        9             Partition Control Messages
        A             General Logging Messages

• Inventors
  Miller, John A.; Svenkeson, Penny L.; Tucker, Brett W.; Erickson, Philip J.; Wilson, Peter C.;
• Assignee
  Unisys (Blue Bell, PA)
• Published
  Aug-31-2004
• Current US Classes:
  712/203
  717/149
  719/313
• Application #
  602791
• International Classes
  G06F 009/46
• Field of Search
  709/312 709/313 709/318 709/327 713/1 713/2 710/260 710/266 710/315 712/28 712/203 719/312 719/313 719/318 719/327 717/149
• Examiner
  Follansbee; John
• Agent
  Atlass; Michael B., Starr; Mark T.
• US Patent References:
  5325517
  5555420
  5724527
  6216216
  6339808
  6370606