Method for managing multiple versions of multiple subsystems in a distributed computing environment

Abstract

A method for use in a parallel distributed computing system having a plurality of processors connected in a network of nodes, each node having software installed thereon, and a control workstation controlling the nodes in the network. A list of the levels of the software installed at each node is stored at the control workstation. A list of software subsystems affected by a command to be executed is stored at the nodes of the network, including the control workstation. In addition, a control script for each of said software subsystems is stored at the nodes, including the control workstation. Each control script provides a routine to be followed for the associated subsystem on the associated node or control workstation during the execution of the command. Since the control scripts are compatible with the level of software installed on the target node, the command will be performed such as to be compatible with the software installed on the target node.

Claims

Having thus described our invention, what we claim as new, and desire to secure by Letters Patent is:

1. In a computing system having a plurality of processors connected in a network of nodes and a control workstation for managing application software to be run on said nodes, a method of executing a command comprising the steps of:

a) sending a command from the control workstation to at least one node for execution by said the processor at said node, said command including a flag indicating the function to be performed by said command;

b) receiving said command at said one node;

c) locating a control script at said one node to perform the function indicated by said flag in said received command, said control script dependent on the level of the application software installed on said one node; and

d) executing the control script by the processor at said one node, thereby performing the function indicated by the flag included in the received command at said one node for the level of application software installed at said one node.

2. The method of claim 1 further comprising storing at said control workstation, a list of the levels of the application software installed at each of said nodes, and wherein step a) includes determining the flag to be included in said command dependent on the level of application software installed on the node to which the command is sent.

3. The method of claim 1 wherein said control script performs a no operation for any function that is not supported by the application software installed at said one node.

4. In a computing system having a plurality of processors connected in a network of nodes, each node having software installed thereon, a method of executing a command at one of the nodes comprising:

storing at said one node, a list of software subsystems affected by the command to be executed;

storing at said one node, a control script for each of said software subsystems, said control script providing a routine to be followed during the execution of the command, said control scripts being compatible with the level of software installed at said one node; and

executing the command at said one node, said command comprising;

locating in said list of software subsystems those software subsystems affected by said command,

locating the control script for each subsystem affected by said command, and

executing the located control script for each subsystem affected by said command, whereby the execution is performed in a manner which is compatible with the level of software installed at said one node.

5. The method of claim 4 further comprising the step of including in said command to be executed, a flag specifying a task to be performed during the execution of said command.

6. The method of claim 5 wherein said control script performs a no operation when the task specified by said flag is not supported by the level of software installed at said one node.

7. In a computing system having a plurality of processors connected in a network of nodes and application software installed on each of said nodes, said application software installed on said nodes being at differing levels, a method of executing a command at one or more nodes comprising:

providing a common interface to the software levels in said network of nodes;

providing at least one control script at each node, said control script being compatible with the level of software on its associated node; and

performing a task through said common interface at one of said nodes, said task being performed via the control script at said one node whereby said task is performed in a manner compatible with the level of software installed at said one node.

8. The method of claim 7 wherein said interface is a command having a flag which indicates the function to be performed by the command, said method further comprising:

executing said command with a flag at said one node; and

performing the function of said flag at said one node in accordance with said control script at said one node, whereby said execution of said command is compatible with the level of software installed at said one node.

9. The method of claim 8 wherein a list of software subsystems affected by said command is recorded at said one node, each of said listed subsystems being linked to a control script, said method further comprising selecting at least one of said software subsystems dependent on said flag, and executing the control script for the selected software subsystem.

10. The method of claim 9 wherein said control script of the selected software subsystem executes a no operation when the function indicated by said flag is not supported by the level of software installed at said one node.

11. In a parallel distributed computing system having a plurality of processors connected in a network of nodes, each node having software installed thereon, and a control workstation controlling the nodes in said network, a method of executing a command at one of the nodes comprising:

storing at said control workstation, a list of the level of the software installed at each node;

sending to said one node, a command to be executed by said one node;

storing at said one node, a list of software subsystems affected by the command to be executed;

storing at said one node, a control script for each of said software subsystems, said control script providing a routine to be followed during the execution of the command, said control scripts being compatible with the level of software installed at said one node; and

executing the command at said one node, said command comprising;

locating in said list of software subsystems those software subsystems affected by said command,

locating the control script for each subsystem affected by said command, and

executing the located control script for each subsystem affected by said command, whereby the execution is performed in a manner which is compatible with the level of software installed at said one node.

12. The method of claim 11 further comprising displaying said list of the level of software installed at each node as stored at said control node.

13. The method of claim 12 wherein said displaying comprises one of; displaying the software level for a particular node, displaying the earliest software level for a group of nodes, and displaying the latest software level for a group of nodes.

14. The method of claim 13 wherein the nodes in computing system are arranged in at least one partition, and said displaying the earliest and latest software levels of a group of nodes comprises displaying the earliest and latest software levels of a system partition.

15. The method of claim 11 further comprising the step of including in said command to be executed, a flag specifying a task to be performed during the execution of said command, said task selected to be compatible with the level of software installed at said one node as determined by said workstation from its list of levels.

16. The method of claim 15 further comprising:

updating the level of the software stored in said list of software levels for said one node; and

sending a command to the other nodes in the network with an appropriate flag notifying said other nodes that the level of said software of said one node is to be changed.

17. The method of claim 15 wherein said control script performs a no operation when the task specified by said flag is not supported by the level of software installed at said one node.

18. In a parallel distributed computing system having a plurality of processors connected in a network of nodes, application software installed on each of said nodes, said application software installed on said nodes being at differing levels, and a control workstation connected to said network for controlling said nodes, a method of executing a command at one or more nodes comprising:

providing a common interface to the software levels in said network of nodes;

providing at least one control script at each node, said control script being compatible with the level of software on its associated node and

sending a task from the control workstation to said one node; and

performing said task through said common interface at one of said nodes, said task being performed via the control script at said one node whereby said task is performed in a manner compatible with the level of software installed at said one node.

19. The method of claim 18 wherein a list is stored at said control workstation listing the level of the software installed at each of the nodes in said network, and said interface is a command having a flag which indicates the function to be performed by the command, said method further comprising:

selecting said flag to be compatible with the software installed at said one node;

executing said command with said flag at said one node; and

performing the function of said flag at said one node in accordance with said control script at said one node, whereby said execution of said command is compatible with the level of software installed at said one node.

20. The method of claim 19 further comprising displaying said list of the level of software installed at each node as stored at said control node.

21. The method of claim 20 wherein said displaying comprises one of; displaying the software level for a particular node, displaying the earliest software level for a group of nodes, and displaying the latest software level for a group of nodes.

22. The method of claim 21 wherein nodes are arranged in at least one partition and said displaying the earliest and latest software levels of a group of nodes comprises displaying the earliest and latest software levels of a system partition.

23. The method of claim 22 further comprising:

updating the level of the software stored in said list of software levels for said one node; and

sending a command to the other nodes in the network with an appropriate flag notifying said other nodes that the level said software of said one node is to be changed.

24. The method of claim 19 wherein a list of software subsystems affected by said command is recorded at said one node, each of said listed subsystems being linked to a control script, said method further comprising selecting at least one of said software subsystems dependent on said flag, and executing the control script for the selected software subsystem.

25. The method of claim 24 wherein said control script of the selected software subsystem executes a no operation when the function indicated by said flag is not supported by the level of software installed at said one node.

Description

BACKGROUND OF THE INVENTION

The present invention relates to parallel, distributed processing systems, and more particularly relates to managing multiple subsystems being run under multiple levels of operating systems and system support software in a parallel, distributed processing system.

A parallel, distributed computing system, such as the IBM RISC/System 6000 Scalable POWERparallel Systems SP computer, contains multiple host computer systems or nodes, each running the AIX operating system (the IBM version of the UNIX operating system) and the IBM Parallel System Support Programs (PSSP) package of system support software. The nodes are grouped into one or more system partitions, each of which represents a logical domain which portrays a virtual SP environment. Introduced with PSSP version 2.2, each node in a system partition may run any supported level of operating system software (AIX). In addition, each node in the system partition may run any level of support software (PSSP) supported by the level of operating system running on that node.

This ability to run different levels of operating system and support software within the same system partition presents a number of new management problems for the administration of the system. Because different levels of operating system and support software may be running across nodes in the same system partition, it must be determined which level(or in some cases which multiple levels) of that support subsystem to initiate on any given node. Since different levels of subsystems must be managed according to the level of underlying software, it becomes administratively tedious and time-consuming for the administrator if he is required to manually track which level is being started and managed on each node. A uniform mechanism (identical interface) is required to automatically start, stop, refresh, etc. a subsystem running on a node regardless of what level of operating system or support software is running on that node.

U.S. Pat. No. 5,367,686 issued Nov. 22, 1994 to Fisher et al. for METHOD FOR AUTOMATED COMPLEX MULTILEVEL SOFTWARE INSTALLATION IN A DATA PROCESSING SYSTEM discloses a method for automatically installing multi-level software in an unattended fashion by building an installation package on a portable storage media. The package contains the data, instructions and pre-selected configuration information to perform the installation without manual intervention.

U.S. Pat. No. 5,444,851 issued Aug. 22, 1995 to Woest for METHOD OF ACCESSING CONFIGURED NODES IN A FACILITIES MANAGEMENT SYSTEM WITH A NON-CONFIGURED DEVICE discloses automated facilities management systems (FMS) wherein different nodes in a distributed FMS are running at different levels of software and/or need to be updated to different levels of software.

U.S. Pat. No. 5,471,617 issued Nov. 28, 1995 to Farrand et al. for COMPUTER MANAGEMENT SYSTEM AND ASSOCIATED MANAGEMENT INFORMATION BASE discloses a method of managing a plurality of networked manageable devices from a manager console using a management information base.

U.S. Pat. No. 5,550,980 issued Aug. 27, 1996 to Pascucci et al. for NETWORKED FACILITIES MANAGEMENT SYSTEM WITH OPTICAL COUPLING OF LOCAL NETWORK DEVICES discloses a controller which provides a unified approach to manage heterogeneous hardware devices. The disclosed facilities management system can automatically trigger commands to be run by sensing some registered change.

U.S. Pat. No. 5,416,917 issued May 16, 1995 to Adair et al. for HETEROGENOUS DATABASE COMMUNICATION SYSTEM IN WHICH COMMUNICATING SYSTEMS IDENTIFY THEMSELVES AND CONVERT ANY REQUESTS/RESPONSES INTO THEIR OWN DATA FORMAT discloses a method and system for establishing the context in which data exchanged between heterogeneous relational DBMS can be mutually understood and preserved and data conversions minimized. In the disclosed system, descriptive information for each machine in the system is stored into a database.

SUMMARY OF THE INVENTION

The disclosed embodiment involves three parts: recording the software level information, reporting (retrieving) that level information, and uniform management of a subsystem regardless of the operating system, support software or subsystem level. To facilitate overall system management of the individual nodes, the operating system level and support software level are recorded in a central repository for general access throughout the system. This information is placed in the central repository via user-executed commands. In order to effectively manage the system partition which may include nodes running multiple levels of the operating system and support software, a new command called "splst.sub.-- versions" is provided to analyze the software level information about one or more nodes in the system partition as recorded in the central repository, and report that information to the requester.

This command will optionally list:

the operating system level of one or more nodes,

the support software level of one or more nodes,

the lowest (earliest) operating system level running within the system partition,

the lowest (earliest) support software level running within the system partition,

the highest (latest) operating system level running within the system partition, and

the highest (latest) support software level running within the system partition.

Support subsystems running within a system partition typically require the running of one or more subsystem daemons on each node to provide services to/from that node to/from the other nodes in the system partition. The software level of the subsystem daemon is usually dictated by a combination of the level of operating system and support software running on the node. On the control node, a subsystem daemon for each different level or subsystem daemon supporting different levels, must be run in order to communicate with the different levels of subsystem daemons running throughout the system partitions.

A new command called the "Syspar Controller" (syspar.sub.-- ctrl) provides the interface to manage the starting, stopping, refreshing, etc. of subsystem daemons throughout the nodes, including the control node. A file called "syspar.sub.-- subsystems" is included and is read by the Syspar Controller. The "syspar.sub.-- subsystems" file contains the list of software subsystem names under control of the Syspar Controller along with the pathname to an associated control script for each subsystem. There is a different syspar.sub.-- subsystems file for each level of support software. The Syspar Controller is invoked with a function code (option flag) by the administrator or by an administrative function of the support software as necessary. The Syspar Controller invokes the control script for each subsystem under its control (as found in the syspar.sub.-- subsystems file) passing the function code along for interpretation by each subsystems' control script. The control script for each subsystem is responsible for performing whatever processing is required to implement that function in the subsystem throughout the system. In this way, the administrator is relieved of the responsibility of managing the potentially many and different levels of subsystem daemons with their varying options throughout the collection of nodes which comprise the system partition.

As mentioned, the Syspar Controller completes its tasks by invoking a control script for each subsystem listed in the syspar.sub.-- subsystems file. This control script is a program which is specific to each different subsystem. (Typically, a control script controls one daemon. However, for some subsystems, it may control more than one.) The control script has intimate knowledge of the interface requirements of its subsystem--how to start it, stop it, refresh it, etc. Every control script (regardless of subsystem daemon) is invoked by the Syspar Controller with the same function code to accomplish the same task (e.g., -s for start, -k for stop, -r for refresh, etc.). The control script turns the generic function code into the specific command invocation required to perform that function on the level of subsystem daemon running on the node or control node. This mechanism relieves the administrator from issuing individual (and potentially different) commands throughout nodes in a system partition when a function is required to be performed on them. The administrator need only ensure that the correct level information has been recorded in the central repository.

It is thus a primary object of the present invention to provide a method for providing a single, uniform interface to manage system functions throughout a system of nodes running mixed levels of software.

It is another object of the present invention to provide a method for managing from a control workstation, system functions on nodes running mixed levels of software of a system having multiple nodes connected in a network of nodes.

It is another object of the present invention to provide a method for displaying the levels of software installed on nodes connected in a network of nodes to assist in the managing of system functions on the nodes.

It is another object of the present invention to provide a method for reporting the changing of the level of software installed on a node in a system of nodes connected in a network, and for allowing for the changing of the level of software on one node without rebooting the software installed on the other nodes.

The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular description of the preferred embodiment of the invention as illustrated in the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a distributed computing system usable with the present invention, the distributed computing system having a control workstation (CWS), multiple nodes and a direct access storage device having stored thereon system data repository (SDR) files;

FIG. 2 is a block diagram showing a syspar.sub.-- subsystems file which lists the subsystems managed by a syspar controller command and control scripts files in the CWS and each node in a partition of nodes of the distributed computing system of FIG. 1;

FIG. 3 is a block diagram showing the operation of the splst.sub.-- versions command of the present invention;

FIG. 4 is a block diagram illustrating the relationship between the syspar controller command, the syspar.sub.-- subsystems file, and the control scripts;

FIG. 5 is a flowchart showing a scenario wherein a new level of operating system and/or support software is introduced into the distributed system, or when the operating system and/or support software level is changed on one or more distributed systems;

FIG. 6 is a flowchart for scenario 1 for examining the syspar.sub.-- subsystems file by the Syspar Controller command;

FIGS. 7a and 7b, when joined at a--a, form a flowchart for scenario 2a for performing a new install of an operating system and support software on the control workstation of the system of FIG. 1;

FIGS. 8a, 8b and 8c, when joined at b--b and c--c, form a flowchart for scenario 2b for performing a new install of an operating system and support software on a node of the system of FIG. 1;

FIGS. 9a, 9b, 9c and 9d, when joined at d--d, e--e and f--f, form a flowchart for scenario 3a for performing a system partition change on the control workstation;

FIGS. 10a, 10b and 10c, when joined at g--g and h--h, form a flowchart for scenario 3b for performing a system partition change on a node;

FIGS. 11a, 11b and 11c, joined at i--i and j--j, form a flowchart for scenario 4a for migrating the control workstation to the latest level of the operating system and support software;

FIGS. 12a, 12b, 12c, 12d and 12e, joined at k--k, l--l, m--m and n--n, form a flowchart for scenario 4b for migrating a node to the latest level of the operating system and the support software; and

FIGS. 13a and 13b, joined at o--o, form a flowchart for scenario 5 for restoring a previously archived SDR and PSSP partitioning environment.

DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention is also described in U.S. patent application Ser. No. 08/898,450 filed Jul. 22, 1999 for MANAGING MULTIPLE VERSIONS OF MULTIPLE SUBSYSTEMS IN A DISTRIBUTED COMPUTING ENVIRONMENT owned by the assignee of the present invention and incorporated herein by reference.

FIG. 1 is a schematic diagram of a distributed computer system 100 useable with the present invention. The distributed computer system 100 may be an IBM RISC System/6000 Scalable POWERparallel Systems (SP) distributed computer system available from International Business Machines Corporation of Armonk, N.Y. The embodiment disclosed in FIG. 1 is an SP computer having a total of 8 frames, with each frame having up to 16 nodes, for a total of 128 nodes. All of the nodes 106 are joined by a local area network (LAN) 102. Each node 106 is a computer itself, and may be a RISC System/6000 workstation, as is well known by those skilled in the art. It will be understood that a greater or lesser number of nodes 106 than the number shown in the present embodiment may be used, if desired.

All of the nodes 106 in a frame of the SP computer 100 are included in a LAN segment which is joined by the other LAN segments through LAN gates 104. Also connected to the LAN 102, is a control workstation (CWS) 112 which controls operation of the SP computer 100. The control workstation has a direct access storage device (DASD) 114 on which is stored, among other files, the system data repository files (SDR). The SDR files include such information as a list of nodes that are in the system and their configuration, and, as will be discussed further, the level of operating system and support software running on each node. Each node 106 also includes a DASD device 107 for storing data processed by the SP computer 100.

In one embodiment, the nodes in each frame are also connected to an IBM Scalable POWERparallel switch (SPS) 105. Each of the SPS switches 105 of each frame is connected to neighboring SPS switches 105 of other frames by a bus 110.

As is well understood in the art, the CWS node 112 may send system data and control signals to the frames of the SP computer 100 by means of the LAN 102 while messages and data may be sent from one to another of the nodes 106 by means of the high performance switches 105.

FIG. 2 is an illustration of a portion of the system of FIG. 1. Shown in FIG. 2, is the control workstation 112 connected to a portion of the nodes 106 in a partition 202 of the SP machine of FIG. 1. Shown in the partition 202, for illustration purposes, are nodes 1, 2, 3, . . . n. The control workstation 112 functions as manager of the distributed computing system and is the interface through which most administrative tasks are performed. The distributed system nodes 106 may run different levels of operating system software (AIX) and support software (PSSP). Software subsystems (such as haem, hags and hats) run on each distributed system and communicate among themselves and the control workstation 112 to provide inter-node services. These software subsystems are well known in the art, and will not be discussed in any further detail. The distributed system 100 runs a set of software subsystems at a level appropriate for the level of operating system and support software running on each node 106. The control workstation 112 may run multiple sets of support subsystems--one for each level represented in the distributed system 100. The System Data Repository (SDR) files 200 which reside on the DASD 114 include a levels data file illustrated at 201. The levels data file 201 includes a listing of the levels of the AIX operating system and PSSP support software being run on each node in the partition 202.

An example of the levels file 201 is shown in Table 1.

                  TABLE 1
    ______________________________________
    Node         AIX Level
                          PSSP Level
    ______________________________________
    1            AIX-414  PSSP-2.1
    2            AIX-420  PSSP-2.2
    3            AIX-325  PSSP-1.2
    .            .        .
    .            .        .
    .            .        .
    n            AIX-414  PSSP-2.2
    ______________________________________

    ______________________________________
    splst.sub.-- versions
    Purpose
    splst.sub.-- versions--Returns information about the
    PSSP code version installed on nodes in the
    SP system.
    Syntax
    n node.sub.-- num!ons ›-G! ›
    h! ›node.sub.-- group! ›
    Flags
    G  Causes the command to look at all system
    partitions rather than just the current
    system partition (but not the control
    workstation).
    1  Returns the latest PSSP version for the
    nodes that are the target of the
    command.
    e  Returns the earliest PSSP version for
    the nodes that are the target of the
    command.
    n node_.sub.-- num
    Returns the PSSP code version for
    node.sub.-- num. Use node.sub.-- num O to specify the
    control workstation.
    N node.sub.-- group
    Returns a list of PSSP versions for
    node.sub.-- group. If -G is supplied, a global
    node group is used. Otherwise, a
    partitioned-bound node group is used.
    t  Returns the node number and PSSP version
    in two columns.
    h  Displays usage information.
    ______________________________________

    ______________________________________
    1.        To list each PSSP version represented in
              the current system partition, enter:
              prompt> splst.sub.-- versions
              PSSP-1.2
              PSSP-2.2
    2.        To list each node in the system
              partition and its PSSP code version,
              enter:
              prompt> splst.sub.-- versions -t
              1 PSSP-2.1
              5 PSSP-2.1
              6 PSSP-2.1
              9 PSSP-2.2
    3.        To list the earliest and latest PSSP
              code versions in a system partition,
              enter:
              prompt> splst.sub.-- versions -1 -e
              PSSP-2.1/* this case has mixed
              partitions */
              PSSP-2.2
    ______________________________________

    ______________________________________
    prompt> splst.sub.-- versions -1 -e
    PSSP-2.2/* this case has only 2.2
    in partition */
    ______________________________________

                  TABLE 2
    ______________________________________
    hats  /usr/lpp/ssp/bin/hatsctrl
    hb  /usr/lpp/ssp/bin/hbctrl
    hags  /usr/lpp/ssp/bin/hagsctrl
    haem  /usr/lpp/ssp/bin/haemctrl
    hr  /usr/lpp/ssp/bin/hrctrl
    pman  /usr/lpp/ssp/bin/pmanctrl
    emon  /usr/lpp/ssp/bin/emonctrl
    sp.sub.-- configd  /usr/lpp/ssp/bin/sp.sub.-- configdctrl
    emcond  /usr/lpp/ssp/bin/emconditionctrl
    spdmd  /usr/lpp/ptpe/bin/spdmdctrl
    ______________________________________

    ______________________________________
    syspar.sub.-- ctrl
    Purpose
    syspar.sub.-- ctrl - Starts, stops, adds, deletes, and
    refreshes the system partition-sensitive
    subsystems installed on the SP system.
    Syntax
    o.vertline..  ctrl ›-G! ›
    R} ›subsystem.sub.-- name!
    ______________________________________

    ______________________________________
    >syspar.sub.-- ctrl -E
    Syspar controller managed subsystems and control
    scripts:
    hats  /usr/lpp/ssp/bin/hatsctrl
    hb  /usr/lpp/ssp/bin/hbctrl
    hags  /usr/lpp/ssp/bin/hagsctrl
    haem  /usr/lpp/ssp/bin/haernctrl
    hr  /usr/lpp/ssp/bin/hrctrl
    pman  /usr/lpp/ssp/bin/pmanctrl
    emon  /usr/lpp/ssp/bin/emonctrl
    sp.sub.-- configd  /usr/lpp/ssp/bin/sp.sub.-- configdctrl
    emcond  /usr/lpp/ssp/bin/emconditionctrl
    spdmd  /usr/lpp/ptpe/bin/spdmdctrl
    ______________________________________

• Inventors
  Beals, Stephanie A.; Briskey, Kenneth C.; Russell, Richard C.;
• Assignee
  International Business Machines Corporation (Armonk, NY)
• Published
  Sep-21-1999
• Current US Classes:
  709/201
  717/170
• Application #
  898609
• International Classes
  G06F 013/00
• Field of Search
  395/200.31 395/712 709/201
• Examiner
  Lim; Krisna
• Agent
  Gonzalez; Floyd A.
• US Patent References:
  5359730
  5367686
  5410651
  5416917
  5421009
  5444851
  5471617
  5550980
  5555416