Method and data processing system providing file I/O across multiple heterogeneous computer systems

Abstract

Bulk data is read or written by an application on a first computer system to a file on a second heterogeneous computer system. Alternatively it is read or written as bulk data directly between applications on these heterogeneous systems. Jobs or tasks are started from one system to execute on a second heterogeneous system. Results are then returned to the first system. Checkpointing and later restarting is also initiated from a first system for execution on the second heterogeneous system.

Claims

What is claimed is:

1. A method of accessing a first file on a disk system on one of a plurality of computer systems from a program executing on another of the plurality of computer systems, wherein:

the plurality of computer systems comprises:

a first computer system containing the program communicating through an API with a first interface system, and

a second computer system containing the disk system and a second interface system for communicating with the first interface system and for reading from and writing to the disk system;

the first computer system and the second computer system are heterogeneous computer systems having different file formats and word structures;

said method comprising:

A) opening a first session from the program via the API through the first interface system to the second interface system in order to access the first file on the disk system;

B) blocking via the API, the first plurality of records into a first plurality of blocks wherein the first plurality of blocks are to be written as a portion of the first file;

C) transmitting the first plurality of blocks over the first session from a first one of the plurality of computer systems to a second one of the plurality of computer systems;

D) unblocking the first plurality of blocks into a second plurality of records on the second one of the plurality of computer systems; and

E) automatically closing the first session after completing the transmitting in step (C).

2. The method in claim 1 wherein:

the first computer system is the first of the plurality of computer systems;

the second computer system is the second of the plurality of computer systems; and

the method further comprises:

F) receiving the first plurality of records via the API from the program; and

G) writing the second plurality of records as a portion of the first file on the disk system.

3. The method in claim 1 wherein:

the first computer system is the second of the plurality of computer systems; and

the second computer system is the first of the plurality of computer systems;

the method further comprises:

F) reading the first plurality of records from the portion of the first file of the disk system; and

G) receiving the second plurality of records in the program via the API.

4. The method in claim 1 wherein:

the transmitting in step (C) utilizes a credit based flow control mechanism to flow control the first plurality of blocks; and

the credit based flow control mechanism utilizes a block based credit counting each of the first plurality of blocks as one credit.

5. The method in claim 1 which further comprises:

F) opening a second session from the program via the API through the first interface system to the second interface system in order to access a second file on the disk system while the first session is still open;

G) blocking via the API, a third plurality of records into a second plurality of blocks wherein the second plurality of blocks is to be written as a portion of the second file;

H) transmitting the second plurality of blocks over the second session from a third one of the plurality of computer systems to a fourth one of the plurality of computer systems;

I) unblocking the second plurality of blocks into a fourth plurality of records on the fourth one of the plurality of computer systems; and

J) closing the second session after completing the transmitting over the second session in step (H).

6. The method in claim 5 wherein:

the first computer system is the first one of the plurality of computer systems and the third one of the plurality of computer systems;

the second computer system is the second one of the plurality of computer systems and the fourth one of the plurality of computer systems; and

the method further comprises:

K) receiving the first plurality of records via the API from the program for transmission over the first session;

L) receiving the third plurality of records via the API from the program for transmission over the second session;

M) writing the second plurality of records to the portion of the first file; and

N) writing the fourth plurality of records to the portion of the second file.

7. The method in claim 5 wherein:

the first computer system is the first one of the plurality of computer systems and the fourth one of the plurality of computer systems;

the second computer system is the second one of the plurality of computer systems and the third one of the plurality of computer systems; and

the method further comprises:

K) receiving the first plurality of records via the API from the program for transmission over the first session;

L) writing the second plurality of records to the portion of the first file;

M) reading the third plurality of records from the portion of the second file; and

N) receiving the fourth plurality of records in the program via the API.

8. The method in claim 1 wherein:

the first computer system is a mainframe computer system; and

the second computer system is a UNIX based computer system.

9. The method in claim 1 wherein:

character data is stored in the first computer system in a first one of a plurality of character formats;

character data is stored in the second computer system in a second one of a plurality of character formats; and

the method further comprises:

F) translating at least a portion of each of the records in the first plurality of blocks from one of the plurality of character formats to another one of the plurality of character formats.

10. The method in claim 1 wherein:

integer data is stored in the first computer system in a first one of a plurality of integer formats;

integer data is stored in the second computer system in a second one of a plurality of integer formats; and

the method further comprises:

F) translating at least a portion of each of the records in the first plurality of blocks from one of the plurality of integer formats to another one of the plurality of integer formats.

11. A data processing system having software stored in a set of Computer Software Storage Media for accessing a first file on a disk system on one of a plurality of computer systems from a program executing on another of the plurality of computer systems, wherein:

the plurality of computer systems comprises:

a first computer system containing the program communicating through an API with a first interface system, and

a second computer system containing the disk system and a second interface system for communicating with the first interface system and for reading from and writing to the disk system;

the first computer system and the second computer system are heterogeneous computer systems having different file formats and word structures;

said software comprising:

A) a set of computer instructions for opening a first session from the program via the API through the first interface system to the second interface system in order to access the first file on the disk system;

B) a set of computer instructions for blocking via the API, the first plurality of records into a first plurality of blocks wherein the first plurality of blocks is being written in a portion of the first file;

C) a set of computer instructions for transmitting the first plurality of blocks over the first session from a first one of the plurality of computer systems to a second one of the plurality of computer systems;

D) a set of computer instructions for unblocking the first plurality of blocks into a second plurality of records on the second one of the plurality of computer systems; and

E) a set of computer instructions for automatically closing the first session after completing the transmitting in set (C).

12. The software in claim 11 wherein:

the first computer system is the first of the plurality of computer systems;

the second computer system is the second of the plurality of computer systems; and

the software further comprises:

F) a set of computer instructions for receiving the first plurality of records via the API from the program; and

G) a set of computer instructions for writing the second plurality of records to the portion of the first file.

13. The software in claim 11 wherein:

the first computer system is the second of the plurality of computer systems; and

the second computer system is the first of the plurality of computer systems; the software further comprises:

F) a set of computer instructions for reading the first plurality of records from the portion of the first file; and

G) a set of computer instructions for receiving the second plurality of records in the program via the API.

14. The software in claim 11 wherein:

the transmitting in set (C) utilizes a credit based flow control mechanism to flow control the first plurality of blocks; and

the credit based flow control mechanism utilizes a block based credit counting each of the first plurality of blocks as one credit.

15. The software in claim 11 which further comprises:

F) a set of computer instructions for opening a second session from the program via the API through the first interface system to the second interface system in order to access a second file on the disk system while the first session is still open;

G) a set of computer instructions for blocking via the API, a third plurality of records into a second plurality of blocks wherein the second plurality of blocks constitutes a portion of the second file;

H) a set of computer instructions for transmitting the second plurality of blocks over the second session from a third one of the plurality of computer systems to a fourth one of the plurality of computer systems;

I) a set of computer instructions for unblocking the second plurality of blocks into a fourth plurality of records on the fourth one of the plurality of computer systems; and

J) a set of computer instructions for closing the second session after completing the transmitting over the second session in set (H).

16. The software in claim 15 wherein:

the first computer system is the first one of the plurality of computer systems and the third one of the plurality of computer systems;

the second computer system is the second one of the plurality of computer systems and the fourth one of the plurality of computer systems; and

the software further comprises:

K) a set of computer instructions for receiving the first plurality of records via the API from the program for transmission over the first session;

L) a set of computer instructions for receiving the third plurality of records via the API from the program for transmission over the second session;

M) a set of computer instructions for writing the second plurality of records to the portion of the first file; and

N) a set of computer instructions for writing the fourth plurality of records to the portion of the second file.

17. The software in claim 15 wherein:

the first computer system is the first one of the plurality of computer systems and the fourth one of the plurality of computer systems;

the second computer system is the second one of the plurality of computer systems and the third one of the plurality of computer systems; and

the software further comprises:

K) a set of computer instructions for receiving the first plurality of records via the API from the program for transmission over the first session;

L) a set of computer instructions for writing the second plurality of records to the portion of the first file;

M) a set of computer instructions for reading the third plurality of records from the portion of the second file; and

N) a set of computer instructions for receiving the fourth plurality of records in the program via the API.

18. The software in claim 11 wherein:

the first computer system is a mainframe computer system; and

the second computer system is a UNIX based computer system.

19. The software in claim 11 wherein:

character data is stored in the first computer system in a first one of a plurality of character formats;

character data is stored in the second computer system in a second one of a plurality of character formats; and

the software further comprises:

F) a set of computer instructions for translating at least a portion of each of the records in the first plurality of blocks from one of the plurality of character formats to another one of the plurality of character formats.

20. The software in claim 11 wherein:

integer data is stored in the first computer system in a first one of a plurality of integer formats;

integer data is stored in the second computer system in a second one of a plurality of integer formats; and

the software further comprises:

F) a set of computer instructions for translating at least a portion of each of the records in the first plurality of blocks from one of the plurality of integer formats to another one of the plurality of integer formats.

21. A computer readable Non-Volatile Storage Medium encoded with software for accessing a first file on a disk system on one of a plurality of computer systems from a program executing on another of the plurality of computer systems, wherein:

the plurality of computer systems comprises:

a first computer system containing the program communicating through an API with a first interface system, and

a second computer system containing the disk system and a second interface system for communicating with the first interface system and for reading from and writing to the disk system;

the first computer system and the second computer system are heterogeneous computer systems having different file formats and word structures;

said software comprising:

A) a set of computer instructions for opening via the API, a first session from the program through the first interface system to the second interface system in order to access the first file on the disk system;

B) a set of computer instructions blocking via the API, the first plurality of records into a first plurality of blocks wherein the first plurality of blocks is to be written as a portion of the first file;

C) a set of computer instructions for transmitting the first plurality of blocks over the first session from a first one of the plurality of computer systems to a second one of the plurality of computer systems;

D) a set of computer instructions for unblocking the first plurality of blocks into a second plurality of records on the second one of the plurality of computer systems; and

E) a set of computer instructions for automatically closing the first session after completing the transmitting in set (C).

22. A data processing system having software stored in a set of Computer Software Storage Media for accessing a first file on a disk system on one of a plurality of computer systems from a program executing on another of

the plurality of computer systems, wherein:

the plurality of computer systems comprises:

a first computer system containing the program communicating through an API with a first interface system, and

a second computer system containing the disk system and a second interface system for communicating with the first interface system and for reading from and writing to the disk system;

the first computer system and the second computer system are heterogeneous computer systems having different file formats and word structures;

said software comprising:

A) means for opening via the API, a first session from the program through the first interface system to the second interface system in order to access the first file on the disk system;

B) means for blocking via the API, the first plurality of records into a first plurality of blocks for writing in a portion of the first file;

C) means for transmitting the first plurality of blocks over the first session from a first one of the plurality of computer systems to a second one of the plurality of computer systems;

D) means for unblocking the first plurality of blocks into a second plurality of records on the second one of the plurality of computer systems; and

E) means for automatically closing the first session after completing the transmitting in means (D).

Description

CROSS REFERENCE TO RELATED APPLICATION

This application is related to our copending patent application entitled Ser. No. 09/896700 "METHOD AND DATA PROCESSING SYSTEM PROVIDING CHECKPOINT/RESTART ACROSS MULTIPLE HETEROGENEOUS COMPUTER SYSTEMS", filed of even date herewith and assigned to the assignee hereof.

This application is related to our copending patent application entitled Ser. No. 09/896037 "METHOD AND DATA PROCESSING SYSTEM PROVIDING REMOTE PROGRAM INITIATION AND CONTROL ACROSS MULTIPLE HETEROGENEOUS COMPUTER SYSTEMS", filed of even date herewith and assigned to the assignee hereof.

This application is related to our copending patent application entitled Ser. No. 09/896702 "METHOD AND DATA PROCESSING SYSTEM PROVIDING BULK RECORD MEMORY TRANSFERS ACROSS MULTIPLE HETEROGENEOUS COMPUTER SYSTEMS", filed on even date herewith and assigned to the assignee hereof, which issued as U.S. Pat. No. 6.615.217 on Sep. 2. 2003.

This application is related to our copending patent application entitled Ser. No. 09/896699 "METHOD AND DATA PROCESSING SYSTEM PROVIDING DATA CONVERSION ACROSS MULTIPLE HETEROGENEOUS COMPUTER SYSTEMS", filed of even date herewith and assigned to the assignee hereof.

FIELD OF THE INVENTION

The present invention generally relates to interconnected heterogeneous data processing systems, and more specifically to reading and writing files by an application on a first system to a disk on a heterogeneous second system.

BACKGROUND OF THE INVENTION

FIG. 1 is a block diagram illustrating a General Purpose Computer 20 in a data processing system. The General Purpose Computer 20 has a Computer Processor 22, and Memory 24, connected by a Bus 26. Memory 24 is a relatively high speed machine readable medium and includes Volatile Memories such as DRAM, and SRAM, and Non-Volatile Memories such as, ROM, FLASH, EPROM, and EEPROM. Also connected to the Bus are Secondary Storage 30, External Storage 32, output devices such as a monitor 34, input devices such as a keyboard 36 (with mouse 37), and printers 38. Secondary Storage 30 includes machine-readable media such as hard disk drives (or DASD) and disk sub-systems. External Storage 32 includes machine-readable media such as floppy disks, removable hard drives, magnetic tapes, CD-ROM, and even other computers, possibly connected via a communications line 28. The distinction drawn here between Secondary Storage 30 and External Storage 32 is primarily for convenience in describing the invention. As such, it should be appreciated that there is substantial functional overlap between these elements. Computer software such as data base management software, operating systems, and user programs can be stored in a Computer Software Storage Medium, such as memory 24, Secondary Storage 30, and External Storage 32. Executable versions of computer software 33, can be read from a Non-Volatile Storage Medium such as External Storage 32, Secondary Storage 30, and Non-Volatile Memory and loaded for execution directly into Volatile Memory, executed directly out of Non-Volatile Memory, or stored on the Secondary Storage 30 prior to loading into Volatile Memory for execution.

FIG. 2 is a block diagram illustrating file reading and writing across heterogeneous systems, in accordance with the Prior Art. In a first computer system 110, an application 120 writes records to a file 114. When the application 120 completes writing to the file 114, the file 114 is closed. Then, a utility, such as FTP, is utilized to transfer the file 114 to a second computer system 112, where a corresponding utility 124 writes the file 116 on disk on that second computer system 112. A second application 126 can then read and process the second file 116. Any necessary translations between the two heterogeneous computer systems is performed by the two utility programs 122, 124.

In the preferred embodiment of this invention, the first computer system 110 is a GCOS® 8 mainframe system that operates utilizing 36-bit words with either 4 9-bit or 6 6-bit characters per word. The preferred second computer system 112 is a UNIX system utilizing 8-bit bytes. The preferred UNIX variant is IBM's AIX. One application that is commonly utilized here is the dumping of a database on the GCOS 8 system 110 to a "flat" file 114. The "flat" file is then moved as bulk data to a Teradata system 112 from NCR, where the "flat" file 114 is loaded into a second database utilizing a "FastLoad" program 126 from NCR.

There are a number of problems with this implementation. Most notably, it is necessary to write the data twice, once on each system, and read it twice, again, once on each system. In large systems, this overhead can be substantial.

FIG. 3 is a block diagram illustrating writing of a file 116 on a second computer system 112 by an application 130 executing on a first computer system 110. The file 116 can then be read and processed by an application 136 on the second computer system 112.

This functionality is available in some homogeneous computer systems. For example, the Solaris operating system sold by Sun provides a Remote File System functionality that allows an application on a first computer system 110 to write files 116 on a second computer system 112. Microsoft Windows (various levels) also supports similar functionality.

However, this functionality has been limited in the prior art to homogeneous computer systems such as Solaris or Windows. It has not been available between heterogeneous computer systems. There are a number of reasons for this. One reason that this functionality has been limited in prior art systems to homogeneous computer systems is that in such cases, there is no requirement to perform any translation between systems, such as between 9 and 8 bit bytes as required in the preferred embodiment of this invention.

FIG. 4 is a block diagram illustrating transferring data directly between an application 130 on a first computer system 110 to a second application 136 on a second computer system 112. This is currently available between applications on heterogeneous computer systems as message passing. One example of a message passing mechanism between heterogeneous computer systems is the FlowBus product sold by the assignee of this invention. However, in the prior art, this is typically fairly slow and expensive due to the requirement to acknowledge messages. It would thus be advantageous to provide this functionality between heterogeneous computer systems utilizing more efficient protocols. In particular, it would be advantageous to provide this functionality for bulk data transfers.

Another problem encountered when utilizing heterogeneous computer systems is that of synchronizing jobs executing on the two computer systems 110, 112. Many variants of UNIX provide the capability to start jobs or tasks on other UNIX systems, to wait for results from the execution of those jobs or tasks, and to receive and act upon those results. However, this capability has not been available in the prior art between heterogeneous computer systems. Some of the problems that have prevented this in the prior art are different formats of data on the two systems, different methods of starting jobs or tasks, and different methods of returning job or task status information. It would thus be advantageous to be able to execute jobs or tasks on a second computer system 112 started from a first heterogeneous computer system 110, which then receives the results of that execution when the jobs or tasks complete.

Another problem encountered when utilizing heterogeneous computer systems is that of checkpointing and restarting jobs or tasks operating on. Again, this feature has been present to some extent when operating across multiple homogeneous computer systems, but not across multiple heterogeneous computer systems. Part of the reason for this problem is that each computer architecture involved utilizes its own unique methods of checkpointing and restarting jobs or tasks. It would thus be advantageous to be able to order checkpointing on a second computer system 112 from a first heterogeneous computer system 110, and then later optionally restarting the checkpointed job or task on that second computer system 112.

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the present invention will be more clearly understood from the following detailed description taken in conjunction with the accompanying Figures where like numerals refer to like and corresponding parts and in which:

FIG. 1 is a block diagram illustrating a General Purpose Computer in a data processing system;

FIG. 2 is a block diagram illustrating file reading and writing across heterogeneous systems, in accordance with the Prior Art;

FIG. 3 is a block diagram illustrating writing of a file on a second computer system by an application executing on a first computer system;

FIG. 4 is a block diagram illustrating transferring data directly between an application on a first computer system to a second application on a second computer system;

FIG. 5 is a block diagram that illustrates in further detail the systems shown in FIG. 3, in accordance with a preferred embodiment of the present invention;

FIG. 6 is a block diagram that illustrates in further detail the systems shown in FIG. 5, in accordance with a preferred embodiment of the present invention;

FIG. 7 is a block diagram illustrating in further detail a channel connected implementation of a preferred embodiment of the present invention;

FIG. 8 is a block diagram illustrating in further detail a communications link connected implementation of a preferred embodiment of the present invention;

FIG. 9 is a block diagram that illustrates in further detail the modules utilized in a preferred embodiment of the present invention; and

FIG. 10 is a flowchart that illustrates the operation of UFAP application, in accordance with a preferred embodiment of the present invention; and

FIG. 11 is a flowchart illustrating the operation of USL software, in accordance with a preferred embodiment of the present invention.

DETAILED DESCRIPTION

Bulk data is read or written by an application on a first computer system to a file on a second heterogeneous computer system. Alternatively it is read or written as bulk data directly between applications on these heterogeneous systems. Jobs or tasks are started from one system to execute on a second heterogeneous system, Results are then returned to the first system. Checkpointing and later restarting is also initiated from a first system for execution on the second heterogeneous system.

Returning to FIG. 3, functionality is described hereinbelow that allows an application 130 on a first computer system 110 to read data from or write data to a file 116 on a second heterogeneous computer system 112. The data is read and/or written as bulk data, in a similar manner as provided by typical file read and writes. An application 136 on the second computer system 112 can then read or write the file 116. The two computer systems 110, 112, may be coupled 132 by a direct channel connection, such as SCSI or Fiber Channel. Alternatively, the two systems may be coupled utilizing communications links and a communications protocol such as TCP/IP. Finally (not shown), the two computer systems 110, 112, may share memory and utilize message passing between the two computer systems 110, 112 for this transfer.

In the preferred embodiment, a program 130 in the first computer system 110 opens one or more files 116 on the second heterogeneous computer system 112. The program 130 then writes to and/or reads from these files 116.

FIG. 5 is a block diagram that illustrates in further detail the systems shown in FIG. 3, in accordance with a preferred embodiment of the present invention. An application 140 executing on the first (mainframe) computer system 110 makes function calls to an Applications Programming Interface (API) 142. Data is written by or read from that API 142 from/to the application 140. The data is then transmitted to 144 or received from 146 the second (UNIX) system 112, where it is written to or read from disk drive(s) 148 on that second system 112.

This provides an efficient mechanism for bulk transfer data from one computer system 110 to files on another heterogeneous computer system 112.

Returning to FIG. 4, functionality is described hereinbelow that allows an application 130 on a first computer system 110 to read data from or write data directly to an application 136 on a second heterogeneous computer system 112. The data is read and/or written as bulk data, in a similar manner as provided by typical file read and writes. When the second computer system 112 is a UNIX system, the application 136 reading or writing the bulk data typically does so utilizing UNIX pipes. These can be coupled (by standard UNIX command language) to 'stdin' and 'stdout'. The two computer systems 110, 112, may be coupled 132 by a direct channel connection, such as SCSI or Fiber Channel. Alternatively, the two coupled utilizing communications links and a communications protocol such as TCP/IP. Finally (not shown), the two computer systems 110, 112, may share memory and utilize message passing between the two computer systems 110, 112 for this transfer.

In the preferred embodiment, a first program 130 in the first computer system 110 starts execution of one or more programs 136 on the second heterogeneous computer system 112. A first pipe is coupled to 'stdin' on each started program 136, and a second pipe is coupled to 'stdout' and 'stderr' on each such started program 136. Then, using the same file read and write interfaces used for remotely reading and writing files in FIG. 3, the first program 130 writes bulk record data that is read by the started program 136 via its 'stdin' file, and reads bulk record data that has been written by the started program 136 on its 'stdout' and 'stderr' files. The two programs 130, 136 receive end-of-file notifications from each other when so indicated. Finally, the started program 136 provides a result or error code and optionally an error string to the first program 130 upon completing. Note that when the second computer system 112 is a UNIX system, all of the standard UNIX utilities are available for remote execution in this manner. Also, as is typical in UNIX, multiple UNIX utilities can be concatenated together using pipes. The first program 130 would thus provide input to the first UNIX program, and receive output from the last in the piped chain. Finally note that a single program 130 on the first system 110 may have multiple files open on the second system 112, as well as pipe connections to multiple started programs on that second system. The same interface is utilized for reading and writing all of such.

FIG. 6 is a block diagram that illustrates in further detail the systems shown in FIG. 5, in accordance with a preferred embodiment of the present invention. An application 140 executing on the first (mainframe) computer system 110 makes function calls to an Applications Programming Interface (API) 142. Data is written by or read from that API 142 from/to the application 140. The data is then transmitted to 144 or received from 146 the second (UNIX) system 112, where it is written to or read from a pipe 147 on that second system 112, where it is read from or written by an application or script 148 on that second system 112.

This provides a mechanism to efficiently transmit bulk record data to/from an application 130 executing on a first computer system 110 from/to an application 136 executing on a second heterogeneous computer system 112.

FIG. 7 is a block diagram illustrating in further detail a channel connected implementation of a preferred embodiment of the present invention. An application 150 on the first (mainframe) computer system 110 is coupled to and communicates with an API 142. The API 142 in turn communicates with a File Interface Library 154. The File Interface Library 154 communicates with a Server Interface Domain (SID) 156. The SID 156 communicates over "Fast Links" 158 such as SCSI or Fiber Channel, to an IO-Manager (IO-MAN) 162 on the second, heterogeneous (UNIX) computer system 112. IO-MAN 162 communicates with a File Interface Library 164 on that system, which in turn either communicates with a File Read/Write utility (UFAP) 166 or via an API (not shown) or via "stdin" with an application 160 on that computer system 112. The File Read/Write utility 166 writes to or reads from a disk(s) on that computer system 112.

FIG. 8 is a block diagram illustrating in further detail a communications link connected implementation of a preferred embodiment of the present invention. An application 150 on the first (mainframe) computer system 110 is coupled to and communicates with an API 142. The API 142 in turn communicates with a File Interface Library 154. The File Interface Library 154 communicates over a communications link 152 utilizing a communications protocol such as TCP/IP with the second, heterogeneous (UNIX) computer system 112. Receiving communications calls on the second computer system 112 is a UNIX Sockets Listener (USL) 168. The USL 168 in turn either communicates with a File Read/Write utility (UFAP) 166 or via an API (not shown) with an application 160 on that computer system 112. The File Read/Write utility 166 writes to or reads from a disk(s) on that computer system 112.

FIG. 9 is a block diagram that illustrates in further detail the modules utilized in a preferred embodiment of the present invention. For the most part, the modules utilized in both heterogeneous computer systems are equivalent, and will be discussed together here. They are discussed more thoroughly below. An application 150 on the first system 110 communicates with a GCOS File Interface Procedure (GFIP) 171. An application 160 and/or the Unix File Read/Write Application (UFAP) 166 on the second system 112 communicates with a UNIX File Interface Procedure (UFIP) 181. Both the GFIP 171 and the UFIP 181 are comprised of an ETL Connection Manager (ECM) 170, 180 which communicate with a Record Manager 172, 182, which in turn either communicate with a SID Interface (SIDI) 174, 184, in the case of a "Fast Link" 158, or a Sockets Interface (SOCKI) 176, 186, in the case of a communications link 152. In the case of a "Fast Link" 158, the SIDI 174 on the first (mainframe) system 110 communicates with the SID 156, which in turn communicates over the "Fast Link" 158 with the IO-MAN 162 on the second system, which in turn communicates with the SIDI 184 on the second system 112. In the case of a communications link 152, the SOCKI 176 routine(s) on the first system 110 communicate with the sockets interface 167, which in turn communicates with the SOCKI routine(s) 186 on the second system 112. The sockets interface 167 also communicates with a UNIX Sockets Listener (USL) 168 which is a UNIX daemon that starts 178 application programs on UNIX 112 when requested to by GCOS 110 clients. The application 160 and/or UFAP 166 on the second system 112 read and write application files on disks 148 on that second system as well as checkpoint/restart files 188 on that system in response to commands received from the first (GCOS) system 110.

The USL 168 starts and monitors jobs or tasks on the second system 112 upon command from the first system 110. Results of the execution of those jobs or tasks are returned to the first system 110. This allows applications 150 on that first system 110 to control execution of applications 160 on the second system 112 in a similar way as applications are controlled in a single system. In the preferred embodiment, both an error code and/or an error message is returned to the application 150 on the first system 110 if the application 160 on the second system 112 fails to execute properly. Testing of the error return can allow the application 150 to determine whether subsequent steps of its execution should be eliminated because of the failure of application 160 to complete its task.

In the case of checkpoint/restart, typically the application 150 on the first system 110 will perform a checkpoint itself and issue a command that the application 160 on the second system 112 also perform a checkpoint. In the case of a UNIX system, this will typically consist of a "flush" followed by recording the position of the file being written or read. Then, if it is necessary to restart the applications 150, 160, the first application 150 with restart itself and rolling back as appropriate and command its peer application 160 to roll back as required. The information from the previous checkpoint needed to synchronize both applications 150, 160 is saved in a restart file 188 on the second system 112.

Another improvement has been made to the prior art. The application 130 on the first computer system 110 can specify what data conversions are to be performed by the interface between systems. Since the data transfers between systems is typically on a (blocked/buffered) record basis, this data conversion can be selected on a per field basis, and is performed on each selected field in each record transferred. Thus, some fields can be converted automatically from 36-bit integers to 32 bit integers (and potentially reversing the "endian" for the integers at the same time), while other fields can be converted from 9-bit ASCII to 8-bit ASCII.

In the preferred embodiment, a "Data Transform Request (DTR) File" (see Appendix A for format of this file) is a parameter to an X_ETL_DEEFINEREC API function call and specifies the conversions that are to be performed. In alternate embodiments, this information is specified by other means, such as by Cobol record formats or a database schema or subschema. Also, in other embodiments, this information can be provided in memory instead of as a file. In the preferred embodiment, this conversion is performed on the first system. However, in other embodiments, this conversion can be performed on the second (UNIX) system.

The preferred embodiment of the present invention consists of a GCOS 8 mainframe computer system as the first computer system 110, and an AIX UNIX computer system as the second computer system 112. It should be understood that this is illustrative only, and that the present invention includes other heterogeneous computer systems.

The remainder of this document describes the design of a product ("Fast ETL") that allows GCOS 8 applications to send a stream of bulk data to a UNIX system connected via normal TCP/IP communication links. GCOS 8 applications are provided with an API that can be accessed via Cobol 85. This API allows data to be streamed both to and from GCOS 8. The same API also allows GCOS 8 applications to stream data to or from a DBSP via its SCSI links. This API allows a GCOS application to open multiple files on a UNIX system. It also allows the GCOS application to start multiple programs on the UNIX system. UNIX pipes are connected to 'stdin', 'stdout', and 'stderr' for each started program. The GCOS application can then read and/or write these UNIX files and pipes interchangeably.

The Fast-ETL system described more fully below is constructed in part utilizing existing Data Base Server Processor (DBSP) code currently being sold by assignee of this invention. More extensive documentation for this product, including documentation for any modules not fully discussed below, is commercially available from assignee upon request. However, this added documentation is not required by persons of ordinary skill in this area of expertise to implement this invention.

1 Overview

1.1 Purpose

This document describes a Fast-ETL system. It discloses a system that allows a mainframe (such as GCOS® 8 sold commercially by the assignee of this invention) applications to send a stream of data to a UNIX system connected via normal TCP/IP communication links. It provides that the system provides an API that is accessible via Cobol 85, and that mainframe applications also be allowed to receive a stream of data. Furthermore, this disclosure provides the same data transmission capability with a DBSP via SCSI links.

The UNIX application that is streaming data with the mainframe (GCOS) application may be either a standard UNIX command, a user-written application, or an application provided with the Fast-ETL product. Assignee's application either reads or writes a UNIX file of standard UNIX file format, thereby allowing the mainframe application to either read or write a standard UNIX file. In the other cases, the Fast-ETL system provides a standard UNIX command or the application with a data stream through the UNIX 'stdin' and 'stdout' file descriptors.

1.2 Basic Design Approach

Two APIs are disclosed—a GCOS 8 Cobal 85 API and a UNIX C API. Those APIs are disclosed in more detail below. Run-time procedures are typically bound with the GCOS 8 and UNIX applications using the Fast-ETL service. In the case of Fast-ETL to a normal UNIX system, these procedures use sockets to transmit the data between GCOS 8 and the UNIX system. In the case of Fast-ETL to a DBSP, these procedures use the services of the SID and IO-MAN components of the DBSP product to transfer the data. SID and IO-MAN are enhanced over the existing products from assignee to provide new functions in support of data streaming; these functions are specifically designed to provide better performance for data streaming than the existing message-exchange service.

Since the Fast-ETL system relies upon the services of either sockets or SID to handle data transfers on the mainframe side, the remainder of this document often refers to Fast-ETL as working in the 'sockets environment' or the 'SID environment'. However, a Fast-ETL application may use data streams in both environments. Depending upon the environment being used to support a stream, the stream is referred to as a 'socket stream' or a 'SID stream'.

In general, the services of Fast-ETL are equally applicable to both the SID and sockets environments The exceptions to this rule are noted below, where the SID environment is more restrictive.

2 Architecture (Highest Level Design)

2.1 Description

Returning to FIGS. 7 and 8, FIG. 8 illustrates the major components of Fast-ETL over sockets. The new software components being developed for Fast-ETL are: