Method of sharing memory between an operating system and an application program

Abstract

An improved method for the sharing of data among a file server, a file system, and a network driver executing on a computer is provided. The file system allocates cache buffers to be used in reading from or writing to a file. When reading a file, the file system starts loading the cache buffers with the file data. When a buffer is loaded, the file system signals the file server that a buffer at a certain address is loaded. The file server provides the network driver with the buffer address of the data that is ready to be transported onto a computer network. The transport system then reads the data from the cache buffers and sends the data onto the computer network. When writing a file, the file system allocates cache buffers to receive the file. The file system notifies the file server of the cache buffer addresses. The file server then notifies the network driver of the cache buffer addresses. When the network driver receives the data it loads the data directly into the cache buffers. The file system can then write the data from the cache buffers to the disk.

Claims

We claim:

1. A method in a computer system for concurrent access of a plurality of portions of external data loaded by an operating system for an application program, the computer having computer memory, the method comprising:

A. the following steps performed by an operating system:

receiving a request of an application program for access to a plurality of portions of external data;

initializing a list of pointers with a predefined value, the pointers for pointing to buffers within the computer memory and being accessible by the application program; and

for each of the plurality of portions of the external data,

retrieving the portion of the external data;

storing the retrieved portion in a buffer; and

updating a pointer in the list of pointers to point to the buffer containing the retrieved portion; and

B. the following steps performed by an application program:

for each pointer in the list of pointers,

when the pointer has been updated to point to a buffer containing the retrieved portion, accessing the portion of the external data stored in the buffer pointed to by the updated pointer before the operating system has completed storing all of the portions of the external data in the buffers.

2. The method of claim 1 wherein the application program is a file server.

3. The method of claim 1 wherein the operating system includes a file system that performs the steps of retrieving and storing the portions of the external data.

4. The method of claim 3 wherein the computer memory is a cache memory of the file system.

5. A method of accessing a file stored on a computer storage device, the storage device operatively connected to a computer, the computer having an operating system and a computer memory, the operating system having a file system, the method comprising the steps of:

requesting the file system to load the file into computer memory for access by an application program; and

for each of a plurality of portions of the file, performing the following steps by the file system:

retrieving the portion of the file from the storage device;

storing the retrieved portion of the file in the computer memory; and

updating a pointer to point to the stored portion of the file; and

performing the following steps by the application program:

waiting until the file system updates the pointer; and

concurrently reading the stored portion of the file pointed to by the updated pointer from the computer memory while the file system is performing the steps of retrieving and storing another portion of the file.

6. The method of claim 5, including the additional step of transporting the read portion of the file onto a computer network, wherein the transporting is performed concurrently with the steps of retrieving and storing another portion of the file.

7. A method for transporting data from a storage device onto a computer network, the storage device and the computer network operatively connected to a file server computer having a file system, a file server, and a transport system, the file system having data descriptors and cache buffers, each data descriptor having a pointer to a cache buffer, the method comprising the steps of:

receiving a request to transport data onto the computer network;

performing the following steps by the file system:

selecting a set of data descriptors to be updated as the data is loaded from the storage device into the cache buffers; and

for each of a plurality of portions of the data,

selecting the next data descriptor from the selected set of data descriptors, starting with the first data descriptor;

selecting a cache buffer to receive the portion of the data to be loaded from the storage device;

loading the portion of the data from the storage device into the selected cache buffer; and

updating the selected data descriptor to point to the selected cache buffer;

for each portion of the data loaded, performing the following steps by the file server:

selecting a next data descriptor from the selected set of data descriptors, starting with the first data descriptor;

waiting for the file system to update the selected data descriptor to point to a cache buffer; and

signalling the transport system to transport onto the computer network the portion of the data in the cache buffer pointed to by the selected data descriptor; and

performing the following steps by the transport system:

for each signal to transport a portion of the data in a cache buffer,

reading the portion of the data stored in the cache buffer; and

writing the data onto the computer network to effect the transport of the data.

8. A method for receiving data from a computer network and storing the data on a storage device, the computer network and the storage device operatively connected to a file server computer having a file system, a file server, and a transport system, the file system having data descriptors and cache buffers, each data descriptor having a pointer for pointing to a cache buffer, the method comprising the steps of:

receiving a request to store data received from a computer network onto the storage device;

selecting a set of data descriptors to be updated as the data is loaded from the computer network into the cache buffers;

setting the pointer in each selected data descriptor to point to a cache buffer;

performing the following steps by the transport system:

for each of a plurality of portions of the data,

selecting a next data descriptor from the selected set of data descriptors, starting with the first data descriptor;

receiving the portion of the data from the computer network; and

storing the received portion of the data in a cache buffer pointed to by the selected data descriptor; and

signalling the file server that all the data has been received;

when signalled that all the data has been received, signalling the file system by the file server that the cache buffers contain the received data; and

when signalled that the cache buffer contains the received data, for each portion of the data stored in the cache buffers, performing the following steps by the file system:

selecting a next data descriptor from the selected set of data descriptors, starting with the first data descriptor; and

writing the portion of the data stored in the cache buffer pointed to by the selected data descriptor to the storage device.

9. The method of claim 8 wherein the step of receiving a request includes the receiving of immediate data.

10. The method of claim 9, including the additional step of storing the immediate data on the storage device.

11. A method in a computer system for accessing a plurality of portions of external data loaded by an operating system for an application program, the computer having computer memory, the method comprising the steps of:

performing the following steps by the operating system:

receiving a request of an application program for access to the external data;

retrieving a portion of the external data;

storing the retrieved portion in a buffer;

setting a pointer to the buffer containing the retrieved portion; and

for each remaining portion of the external data,

retrieving a new portion of the external data;

storing the retrieved new portion in a buffer other than the buffer pointed to by the pointer;

waiting until the application program accesses the pointer; and

updating the pointer to point to the buffer containing the retrieved new portion; and

performing the following steps by the application program:

waiting until the operating system sets the pointer to point to the buffer containing the retrieved portion;

when the pointer has been set, accessing the retrieved portion of the external data stored in the buffer pointed to by the set pointer, while the operating system is retrieving a new portion of the external data;

for another portion of the external data, waiting until the operating system updates the pointer to point to the buffer containing the retrieved new portion; and

when the pointer has been updated, accessing the portion of the external data stored in the buffer pointed to by the updated pointer, while the operating system is retrieving a new portion of the external data.

Description

TECHNICAL FIELD

This invention relates generally to a computer system for the sharing of data between a file server and a file system, and more specifically, to a method and an apparatus for sharing of cache memory by a file server and a file system.

BACKGROUND ART

Local Area Networks (LANs) are very popular means for interconnecting computers, especially personal computers. These computers are interconnected so that the users can share resources, such as files. The users of networked computers have found that their productivity is often increased by sharing files. For example, in a law office environment in which attorneys and secretaries have networked computers, an attorney may prepare a draft of a document using a word processor program, such as Microsoft Word, and store the document in a shared file. The attorney's secretary can access the shared file, use the word processing program to put the document in final form, and either print the document or store the revised document in the shared file so that the attorney can make a final review.

Personal computers and their controlling programs, called operating systems, were originally designed to work in a stand-alone mode, that is, not interconnected in a network. The portion of the operating system that manages the files is called the "file system." The designs of file systems, such as the file system of the MS-DOS.RTM. operating system, were simplified because the computer was assumed to be used in stand-alone mode. An operating system that supports file sharing is much more complex because it must ensure that two users are not revising the same file at the same time. Otherwise, one user could revise the file and another user could unintentionally replace those revisions with new revisions, causing the loss of the first revisions.

When personal computers were first interconnected into a network, a primary purpose was to share files. However, the then existing file systems did not support file sharing. To overcome this inability to support file sharing, a "file server" application program was introduced. FIG. 1 shows a network that has three computers 101, 102, 103 interconnected. A file server computer 103 executes the file server application program 103a, which provides shared access to the files on a disk drive 104. When an application executing on one of the other computers 101, 102 requests to read a file on the disk drive 104, the requesting application 101a, 102a calls its local file system 101b, 102b. The local file system 101b, 102b determines that the request is for a file in a remote drive 104 and sends the request to the file server computer 103 through its local network driver 101c, 102c. The network driver 103c for the file server computer 103 receives the request and routes the file request to the file server 103a. The file server 103a ensures that the requesting application 101a, 102a has read access to the file. The file server 103a then calls its local file system 103b to retrieve the data from the file. The file server 103a then sends the data through the network to the requesting computer.

It is very useful to have an efficient file server. Because many computers may be networked together, there may be a large demand for access to shared and nonshared files on a disk drive 104. However, the file systems of prior operating systems were not designed to support file servers that support a network of computers.

When retrieving data from a file, prior systems would move the data several times in the computer memory. Referring to FIG. 2, the file system of prior operating systems would load the data from the file into a buffer 201 in the data area of the file system. The data area of the file system is referred to as the "cache memory" or "cache buffer." The file system would then move the from its buffer 201 to a buffer 202 in the data area of the application. Because files are often very large, this moving from cache buffer to application buffer is very time-consuming.

It is desirable to have a file system and file server in which the move from the cache buffer to the application buffer could be eliminated and still allow the file server to access the data.

DISCLOSURE OF THE INVENTION

It is an object of the present invention to provide a method and system of sharing data between a file server and a file system.

It is another object of the present invention to provide a file server that needs no internal data buffers for storing the files retrieved from the file system.

It is another object of the present invention to provide a method and system for transporting files on a computer network without the need for making multiple copies of the file in the internal memory of the transporting computer.

It is another object of the present invention to provide a method and system for transporting a file on a computer network in an overlap mode in which parts of the file are transported while other parts of the file are being retrieved from the disk.

These and other objects, which will become apparent as the invention is more fully described below, are obtained by providing a method and system of cache sharing between a file server and a file system. In a preferred embodiment, a system according to the present invention accomplishes the sharing by providing a set of application program interfaces (API) to the file system. The APIs return pointers to the data in the cache memory rather than moving the data to the application buffer. The APIs also coordinate the holding of these cache buffers to ensure that the data is present when needed by the file server.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic of a prior computer interconnected to a file server.

FIG. 2 shows the copying of data from a file system to an application in prior systems.

FIG. 3 shows an overview flow chart of a file server and file system of a preferred embodiment of the present invention.

FIG. 4 shows the layout of certain data structures used in a preferred embodiment of the present invention.

FIG. 5 shows the data structure layout of the PNBs, Buffer Descriptors, and Data Descriptors.

FIGS. 6A, 6B, 6C, and 6D are a flow diagram of the subroutine ReadRaw.

FIGS. 7A, 7B, and 7C are a flow diagram of the subroutine ReadInit.

FIG. 8 is a flow diagram of the subroutine RdAsync.

FIG. 9 shows the relationship between the File Handle Table and the Open File Table.

FIG. 10 is a flow diagram of the subroutine ReadFinish.

FIGS. 11A and 11B are a flow diagram of the subroutine WriteRaw.

FIG. 12 is a flow diagram of the subroutine WriteInit.

FIGS. 13A and 13B are a flow diagram of the subroutine SetUpWD.

FIG. 14 is a flow diagram of subroutine WriteFree.

FIG. 15 is a flow diagram of subroutine WriteFinish.

DETAILED DESCRIPTION OF THE INVENTION

In a preferred embodiment of the present invention, the file system, file server, and network driver cooperatively interact to reduce the moving of data and to increase the overlapping of disk I/O and network I/O when a file is transferred from one computer to another. When the file server requests a file read, the file system returns a data structure, called a Data Descriptor List. Each Data Descriptor in the list contains information relating to a cache buffer in which a portion of the file will be stored. The file system then stores the file in one or more cache buffers. When a cache buffer has been loaded with part of the file, the file system puts the address of the cache buffer in the appropriate Data Descriptor. The file server is notified of the storing of the address and then directs the network driver to transport the data that is in the cache buffer. When the transport is complete, the file server then waits until notified that the next cache buffer is ready to transport.

FIG. 3 provides an overview flow chart of the file server and file system in a preferred embodiment of the present invention. The flow chart on the left represents the operation of the file server and the flow chart on the right represents the operation of the file system. In block 301, the file server requests 314 the file system to prepare a list of Data Descriptors for the file to be read. The file system returns 315 a pointer to the Data Descriptors.

In block 307, the file system prepares the Data Descriptor List for the file to be read. In block 308, the file system selects the next Data Descriptor in the list, starting witch the first Data Descriptor in the list. In block 309, the file system selects a cache buffer that is available to be loaded with data. In block 310, the file system issues a read to put the next portion of the file into the selected cache buffer.

In block 311, the file system waits for the read to complete. In block 312, the file system updates the selected Data Descriptor with the address of the selected cache buffer. This signals 316 the file server that the cache buffer is filled. In block 313, if all the Data Descriptors have been processed, then the read is complete, else the file system loops to block 308 to process the next Data Descriptor, that is, read in the next portion of the file.

In blocks 302 through 306, the file server loops processing each Data Descriptor, that is, waiting until the associated cache buffer is filled, transporting the data, and releasing the cache buffer. In block 302, the file server selects the next Data Descriptor in the list, starting with the first Data Descriptor in the list. In block 303, when the file server determines that a cache buffer has been filled for the selected Data Descriptor, it continues at block 304. In block 304, the file server sends a request to the network driver to transport the data stored in the selected cache buffer. In block 306, if all the Data Descriptors have been processed, then the read is complete, else the file server loops to block 302 to process the next Data Descriptor.

In a preferred embodiment of the present invention, the file server and file system also provide a write file capability. The write file capability provides many of the same advantages of the read file capability, such as increasing network I/O and disk I/O overlap and reducing the number of times the file data is moved in memory. A detailed description of the read and write capabilities of a preferred embodiment of the present invention is given following an overview of the file system caching and the data structures used in a preferred embodiment of the present invention.

File System Caching

A typical file system stores data on a disk with a given sector size, such as 512 bytes. Data can only be read from or written to a disk in a full sector, that is, every byte in a sector must be read or written at the same time. A file is not necessarily stored in contiguous sectors on a disk. Each contiguous sector that holds contiguous portions of the file is an extent. Each file has an associated extent table, which contains a list of all the extents that comprise the file.

In prior systems, when an application program requested the file system to read a file, the file system first copied the file into a portion of main memory allocated to the file system, called the file system buffers. The file system then copied the file from its buffers to the buffer of the application program.

In a disk-caching file system, when a read is requested, the file system first checks its buffers, called the cache, to determine whether the requested file has already been transferred from disk to the file system buffers. If so, then the file system does not need to retransfer the file to the cache.

In a preferred embodiment, the file system maintains cache headers and cache buffers. There is a one-to-one correspondence between the cache headers and buffers. In a preferred embodiment, the cache buffers are 2048 bytes in length (2K). Each cache buffer can hold four contiguous sectors of disk data, assuming a 512-byte sector. Each cache header contains information about the associated cache buffer. In a preferred embodiment, the cache header contains the start sector address b.sub.-- sec and a flag b.sub.-- dirt indicating whether the buffer needs to be written to disk. When the file system gets a read request, it first checks the cache header to determine whether the file or a portion of the file is already in a cache buffer. If a portion of the file is in a cache buffer, then a transfer from disk can be eliminated. The variable b.sub.-- addr in the cache header points to the associated cache buffer.

Data Structures

FIG. 4 shows the relationships of several of the data structures used in a preferred embodiment of the present invention. Each cache header 401 has an associated cache buffer 402. Each cache header contains information relating to the disk data that is currently in the associated cache buffer. The cache headers and cache buffers are in the data area of the file system. The Descriptor List 403 is also in the data area of the file system. The file system initializes the Data Descriptor List 403 for each read or write request of the present invention. Each Data Descriptor contains information relating to a portion of the file to be read or written. For each Data Descriptor there is an associated cache header. The Data Descriptors contain pointers to the associated cache header and cache buffer and the sector address of the data in the cache buffer.

The file server uses the data structures PNB 404 and Buffer Descriptors 405. The PNB entries contain a network control block (NCB). The NCB is passed by the file server to the network driver to indicate the type of transport request. Each NCB points to a set of Buffer Descriptors 405, each of which points to an associated cache buffer 402.

Table 1 contains the names of the variables and a brief description of the variables used by a preferred embodiment of the present invention.

                  TABLE 1
    ______________________________________
    Data Structures
                   Description
    ______________________________________
    Data Descriptor
    dd.sub.-- addr points to first byte to transfer in
                   the cache buffer
    dd.sub.-- length
                   number of bytes to transfer in the
                   cache buffer
    dd.sub.-- offset
                   byte offset where the data begins
    dd.sub.-- sector
    SNUM           disk sector address of data
    SHINT          pointer to cache buffer header
    Buffer Descriptor
    bf.sub.-- DataPtr
                   pointer to the first byte in cache
                   buffer to transfer
    bf.sub.-- DataLen
                   number of bytes to transfer
    PNB
    pnb.sub.-- list
                   pointer to link the PNBs
    pnb.sub.-- flag
                   holds internal state
    pnb.sub.-- ncb
    ncb.sub.-- command
                   network control command
    ncb.sub.-- buffer
                   pointer to the Buffer Descriptors
    ncb.sub.-- length
                   count of bytes to transfer
    ncb.sub.-- flags
                   indicates position of NCB group of
                   NCBs
    ncb.sub.-- cmd.sub.-- cplt
                   indicates that the network has
                   completed the network request
                   associated with this NCB
    Read Work Area
    rwk.sub.-- cbData
                   number of bytes in file to read
    rwk.sub.-- pDDesc
                   pointer to next Data Descriptor to
                   send
    rwk.sub.-- cDDesc
                   number of Data Descriptors left to
                   send
    rwk.sub.-- ncbs
                   pointer to list of NCBs
    rwk.sub.-- cNcbs
                   number of NCBs
    rwk.sub.-- bufDesc
                   Buffer Descriptors
    Cache Header
    b.sub.-- sec   sector address of data in cache buffer
    b.sub.-- addr  pointer to cache buffer
    b.sub.-- lcnt  lock count
    b.sub.-- hcnt  hold count
    b.sub.-- dirt  flag indicating whether the cache
                   needs to be written to disk
    OFT
    oft.sub.-- hcnt
                   hold count for an open file
    Write Work Area
    wwk.sub.-- cImm
                   number of bytes of immediate data
    wwk.sub.-- pImm
                   pointer to immediate data
    wwk.sub.-- cRaw
                   count of bytes of raw data that will
                   fit into the last cache buffer of
                   immediate data
    wwk.sub.-- cbytes
                   number of bytes to write to the file
    wwk.sub.-- pDDesc
                   pointer to Data Descriptors
    wwk.sub.-- cDDesc
                   count of Data Descriptors
    wwk.sub.-- bufDesc
                   Buffer Descriptors
    ______________________________________

• Inventors
  Fenwick, Thomas; Rubin, Darryl E.;
• Assignee
  Microsoft Corporation (Redmond, WA)
• Published
  Sep-30-1997
• Current US Classes:
  709/213
  709/234
  711/118
  711/147
  711/3
  719/312
• Application #
  113798
• International Classes
  H01J 013/00
• Field of Search
  395/200 395/275 395/325 395/600 395/575 395/650 395/425 395/250 395/700 395/200.01-200.08 395/200.13 395/440 395/450 395/474-480
• Examiner
  Hafiz; Tariq R.
• Agent
  Seed and Berry LLP
• US Patent References:
  4309755
  4486854
  4881163
  4887204
  4897781
  4956808
  4994963
  5001628
  5032979
  5058006
  5089956
  5093779
  5113519
  5129083
  5133053
  5151990
  5155842
  5163131
  5218695
  5249279
  5301350
  5305440
  5347632
  5355453