Global file system and data storage device locks

Abstract

A system includes shared Small Computer System Interface (SCSI) storage devices for processing clients coupled by a fiber channel interface. The storage devices include storage blocks, and locks controlling their use by clients. Clients issue actions to the storage devices for performing operations on the locks. A client may exclude other clients from using storage blocks using a state element to acquire the lock for shared or exclusive use. If the client modified the data, a version counter in the lock is updated when the lock is released. If an activity bit is set, the version counter is updated upon both reading and writing. Other clients can forcibly release a lock owned by a failed client by monitoring its version counter. Expiration timers associated with the locks allow acquired locks to be released by timing out. A serverless global file system (GFS) manages use of the shared storage resources, and allows remapping of the locks to the storage blocks, for example, based on activity of the locks.

Claims

What is claimed is:

1. A data storage device that is accessible by first and second remote processing clients, the storage device including:

a communication interface that receives control signals from a distributed file system residing on both the first and second remote processing clients; and

at least one lock mechanism, operatively coupled to the communication interface, each lock mechanism associated with the use of a storage block on the storage device by each of the first and second processing clients, wherein each lock mechanism controls access to a particular storage block based on the control signals received from the distributed file system, the first and second remote processing clients connecting to the storage block by bypassing a protocol stack overhead.

2. The device of claim 1, in which each lock mechanism is adapted for allowing access by one of the first and second processing clients to the storage block associated with the lock mechanism, and excluding the other of the first and second processing clients from accessing the storage block associated with the lock mechanism.

3. The device of claim 1, further comprising a version counter, associated with the lock mechanism, the version counter being updated when data is written to the storage block associated with the lock mechanism.

4. A data storage device that is accessible by first and second remote processing clients, the storage device including:

a communication interface that receives control signals from a distributed file system residing on both the first and second remote processing clients;

at least one lock mechanism, operatively coupled to the communication interface, each lock mechanism associated with the use of a storage block on the storage device by each of the first and second processing clients, wherein each lock mechanism controls access to a particular storage block based on the control signals received from the distributed file system; and

a version counter, associated with the lock mechanism, the version counter being updated when data is written to the storage block associated with the lock mechanism, in which the lock mechanism is adapted to be released based on one of (a) a state of the version counter associated with the lock mechanism, and (b) activity of the version counter associated with the lock mechanism.

5. The device of claim 4, further comprising an activity element associated with the lock mechanism, wherein the activity element and the version counter associated with the lock mechanism are each controlled by at least one of the first and second processing clients such that:

if the activity element is not set, the version counter is updated when data is written to the particular storage block controlled by the lock mechanism; and

if the activity element is set, the version counter is updated when data is written to or read from the particular storage block associated with the lock mechanism.

6. The device of claim 4, in which the file system includes a remapping system that remaps the lock mechanisms to the storage blocks based on activity of the version counter associated with the lock mechanism relative to activity of a version counter associated with another lock mechanism.

7. The device of claim 1, further comprising first and second storage media, the first storage medium being faster to access than the second storage medium, wherein the lock mechanism resides on the first storage medium, and the storage block associated with the lock mechanism resides on the second storage medium.

8. The device of claim 7, in which the first storage medium provides volatile storage and the second storage medium provides nonvolatile storage.

9. The device of claim 1, in which the lock mechanism includes an expiration timer.

10. The device of claim 1, in which the lock mechanism includes a list of processing clients that have accessed the lock mechanism.

11. A data storage device that is accessible by first and second remote processing clients, the storage device including:

a communication interface that receives control signals from a distributed file system residing on both the first and second remote processing clients; and

at least one lock mechanism, operatively coupled to the communication interface, each lock mechanism associated with the use of a storage block on the storage device by each of the first and second processing clients, wherein each lock mechanism controls access to a particular storage block based on the control signals received from the distributed file system;

a state element associated with the lock mechanism, an activity element associated with the lock mechanism, a version counter associated with the lock mechanism, and a processor that is adapted for executing an action selected from a group consisting of:

a lock action, initiated by one of the first and second processing clients, the lock action identifying a specific one of the lock mechanisms and, in response to the lock action, the processor acquiring the identified lock mechanism, if available, and communicating whether the identified lock mechanism was acquired to the initiating processing client;

an unlock action, initiated by one of the first and second processing clients, the unlock action identifying a specific one of the lock mechanisms and, in response to the unlock action, the processor releasing the identified lock mechanism and, if the activity element associated with the identified lock mechanism is set, updating the version counter controlled by the identified lock mechanism;

an unlock increment action, initiated by one of the first and second processing clients, the unlock increment action identifying a specific one of the lock mechanisms and, in response to the unlock increment action, the processor releasing the identified lock mechanism and updating the version counter controlled by the identified lock mechanism independently of whether the activity element associated with the identified lock mechanism is set;

a reset lock action, initiated by one of the first and second processing clients, the reset lock action identifying a specific one of the lock mechanisms and, in response to the reset lock action, if a state of the version counter controlled by the identified lock mechanism matches an input version counter value, the processor (a) releases the identified lock mechanism, and (b) updates the version counter controlled by the identified lock mechanism;

an activity on action, initiated by one of the first and second processing clients, the activity on action identifying a specific one of the lock mechanisms and, in response to the activity on action, the processor sets the activity element associated with the identified lock mechanism;

an activity off action, initiated by one of the first and second processing clients, the activity off action identifying a specific one of the lock mechanisms and, in response to the activity off action, the processor clears the activity element associated with the identified lock mechanism;

a lock shared action, initiated by one of the first and second processing clients, the lock shared action identifying a specific one of the lock mechanisms and, in response to the lock shared action, the processor acquiring the identified lock mechanism for shared use by the initiating processing client, if available, and communicating whether the identified lock mechanism was acquired to the initiating processing client;

a lock exclusive action, initiated by one of the first and second processing clients, the lock exclusive action identifying a specific one of the lock mechanisms and, in response to the lock exclusive action, the processor acquiring the identified lock mechanism for exclusive use by the initiating processing client, if available, and communicating whether the identified lock mechanism was acquired to the initiating processing client;

a force lock exclusive action, initiated by one of the first and second processing clients, the force lock exclusive action identifying a specific one of the lock mechanisms that is acquired exclusively by a different one of the first and second processing clients and, in response to the force lock exclusive action, the processor acquiring the identified lock mechanism for exclusive use by the initiating processing client;

a touch lock action, initiated by one of the first and second processing clients, the touch lock action modifying an expiration timer associated with at least one lock mechanism that is acquired by the initiating processing client; and

a report expired action, initiated by one of the first and second processing clients, the report expired action communicating to the initiating processing client a status of at least one expiration timer, the expiration timers being associated with the lock mechanisms.

12. A data storage device that is accessible by first and second remote processing clients, the storage device including:

a communication interface that receives control signals from a distributed file system residing on both the first and second remote processing clients; and

at least one lock mechanism, operatively coupled to the communication interface, each lock mechanism associated with the use of a storage block on the storage device by each of the first and second processing clients, wherein each lock mechanism controls access to a particular storage block based on the control signals received from the distributed file system;

a state element associated with the lock mechanism, an activity element associated with the lock mechanism, a version counter associated with the lock mechanism, and a processor that is adapted for executing an action selected from a group consisting of:

a lock action, initiated by one of the first and second processing clients, the lock action identifying a specific one of the lock mechanisms and, in response to the lock action, the processor acquiring the identified lock mechanism, if available, and communicating whether the identified lock mechanism was acquired to the initiating processing client, wherein if the processor, in response to the lock action, determines that the state element of the identified lock mechanism is set, the storage device communicates a second return result to the initiating processing client, otherwise the processor sets the state element of the identified lock mechanism and the storage device communicates a first return result, which is different from the second return result, to the initiating processing client;

an unlock action, initiated by one of the first and second processing clients, the unlock action identifying a specific one of the lock mechanisms and, in response to the unlock action, the processor releasing the identified lock mechanism and, if the activity element associated with the identified lock mechanism is set, updating the version counter controlled by the identified lock mechanism, wherein the processor, in response to the unlock action, clears the state element of the identified lock mechanism and the storage device communicates a first return result to the initiating processing client, and if the activity element of the identified lock mechanism is set, the processor increments the version counter associated with the identified lock mechanism;

an unlock increment action, initiated by one of the first and second processing clients, the unlock increment action identifying a specific one of the lock mechanisms and, in response to the unlock increment action, the processor releasing the identified lock mechanism and updating the version counter controlled by the identified lock mechanism independently of whether the activity element associated with the identified lock mechanism is set, wherein the processor, in response to the unlock increment action, increments the version counter associated with the identified lock mechanism and clears the state element of the identified lock mechanism, and the storage device communicates a first return result to the initiating processing client;

a reset lock action, initiated by one of the first and second processing clients, the reset lock action identifying a specific one of the lock mechanisms and, in response to the reset lock action, if a state of the version counter controlled by the identified lock mechanism matches an input version counter value, the processor (a) releases the identified lock mechanism, and (b) updates the version counter controlled by the identified lock mechanism, wherein if the processor, in response to the reset lock action, determines that the state of the version counter associated with the identified lock mechanism matches the input version counter value, the processor increments the version counter associated with the identified lock mechanism and clears the state element of the identified lock mechanism, and the storage device communicates a first return result to the initiating processing client, otherwise the storage device communicates a second return result to the initiating processing client;

an activity on action, initiated by one of the first and second processing clients, the activity on action identifying a specific one of the lock mechanisms and, in response to the activity on action, the processor sets the activity element associated with the identified lock mechanism, wherein the storage device, in response to the activity on action, communicates a first return result to the initiating processing client after the processor sets the activity element associated with the identified lock mechanism;

an activity off action, initiated by one of the first and second processing clients, the activity off action identifying a specific one of the lock mechanisms and, in response to the activity off action, the processor clears the activity element associated with the identified lock mechanism, wherein the processor, in response to the activity off action, increments the version counter associated with the identified lock mechanism, and the storage device communicates a first return result to the initiating processing client after the processor clears the activity element associated with the identified lock mechanism;

a lock shared action, initiated by one of the first and second processing clients, the lock shared action identifying a specific one of the lock mechanisms and, in response to the lock shared action, the processor acquiring the identified lock mechanism for shared use by the initiating processing client, if available, and communicating whether the identified lock mechanism was acquired to the initiating processing client;

a lock exclusive action, initiated by one of the first and second processing clients, the lock exclusive action identifying a specific one of the lock mechanisms and, in response to the lock exclusive action, the processor acquiring the identified lock mechanism for exclusive use by the initiating processing client, if available, and communicating whether the identified lock mechanism was acquired to the initiating processing client;

a force lock exclusive action, initiated by one of the first and second processing clients, the force lock exclusive action identifying a specific one of the lock mechanisms that is acquired exclusively by a different one of the first and second processing clients and, in response to the force lock exclusive action, the processor acquiring the identified lock mechanism for exclusive use by the initiating processing client;

a touch lock action, initiated by one of the first and second processing clients, the touch lock action modifying an expiration timer associated with at least one lock mechanism that is acquired by the initiating processing client; and

a report expired action, initiated by one of the first and second processing clients, the report expired action communicating to the initiating processing client a status of at least one expiration timer, the expiration timers being associated with the lock mechanisms.

13. A system including the storage device of claim 1, the system further comprising a network coupling the first and second processing clients to the communications interface of the storage device.

14. A method of using a system having first and second processing clients, a distributed file system that resides on both the first and second processing clients, a data storage device shared by the first and second processing clients, a communication network linking the storage device and the first and second processing clients bypassing protocol stack overheads, the data storage device including storage blocks and lock mechanisms, the method comprising steps of:

(a) assigning a lock mechanism to a storage block using the file system; and

(b) accessing at least one of the storage blocks, using the first processing client, by acquiring the lock mechanism assigned to the storage block, if the lock mechanism is available.

15. The method of claim 14, further comprising a step (c) of releasing the lock mechanism after accessing the storage block.

16. The method of claim 15, in which the step (c) of releasing includes updating a version counter associated with the lock mechanism if an activity element associated with the lock mechanism is set.

17. The method of claim 14, further comprising steps of:

(c) writing data from the first processing client to the storage block;

(d) updating a version counter associated with the lock mechanism using the first processing client; and

(e) releasing the lock mechanism using the first processing client.

18. The method of claim 14, wherein step (b) of accessing includes waiting, if the lock mechanism is unavailable, for the lock mechanism to become available before acquiring the lock mechanism.

19. A method of using a system having first and second processing clients, a distributed file system that resides on both the first and second processing clients, a data storage device shared by the first and second processing clients, a communication network linking the storage device and the first and second processing clients, the data storage device including storage blocks and lock mechanisms, the method comprising steps of:

(a) assigning a lock mechanism to a storage block using the file system;

(b) accessing at least one of the storage blocks, using the first processing client, by acquiring the lock mechanism assigned to the storage block, if the lock mechanism is available;

(c) setting an activity element associated with the lock mechanism; and

(d) updating a version counter associated with the lock mechanism when either of the first and the second processing clients reads data from or writes data to the storage block assigned to the lock mechanism, if the activity element is set.

20. The method of claim 19, further comprising steps of:

(e) waiting for a predetermined period of time after setting the activity element;

(f) updating the version counter associated with the lock mechanism;

(g) releasing the lock mechanism, if the version counter associated with the lock mechanism is not updated during the predetermined period of time; and

(h) clearing the activity element associated with the lock mechanism, if the version counter associated with the lock mechanism is updated during the predetermined period of time.

21. The method of claim 20, in which the step (g) of releasing comprises sending a unit attention to at least one of the first and second processing clients indicating that the lock mechanism has been released.

22. A method of using a system having first and second processing clients, a distributed file system that resides on both the first and second processing clients, a data storage device shared by the first and second processing clients, a communication network linking the storage device and the first and second processing clients, the data storage device including storage blocks and lock mechanisms, the method comprising steps of:

(a) assigning a lock mechanism to a storage block using the file system;

(b) accessing at least one of the storage blocks, using the first processing client, by acquiring the lock mechanism assigned to the storage block, if the lock mechanism is available;

(c) requesting access, using the second processing client, to the storage block for which the first processing client has acquired the lock mechanism;

(d) setting an activity element associated with the lock mechanism, using the second processing client, if the second processing client fails to acquire the lock mechanism;

(e) updating a version counter associated with the lock mechanism when the first processing client reads data from or writes data to the at least one storage block assigned to the lock mechanism, if the activity element is set;

(f) waiting for a predetermined period of time after using the second processing client to set the activity element;

(g) updating the version counter using the second processing client, if the first processing client has not updated the version counter during the predetermined period of time, otherwise, clearing the activity element associated with the lock mechanism using the second processing client if the version counter is updated by the first processing client during the predetermined period of time; and

(h) releasing the lock mechanism using the second processing client, if the first processing client has not updated the version counter during the predetermined period of time.

23. The method of claim 22, in which the step (g) of updating includes providing an input version counter value to the storage device and determining whether a state of version counter associated with the lock mechanism matches the input version counter value.

24. The method of claim 22, further comprising a step (I) of accessing the storage block, using the second processing client.

25. The method of claim 22, wherein the step (h) of releasing comprises sending a unit attention from the storage device to the first processing client indicating that the lock mechanism was released by the second processing client.

26. The method of claim 14, further comprising steps of:

(c) using the second processing client to request access to the storage block for which the first processing client has acquired a lock mechanism;

(d) waiting until the first processing client has released the lock mechanism before using the second processing client to acquire the lock mechanism on the storage device; and

(e) accessing the storage block using the second processing client.

27. The method of claim 26, further comprising steps of:

(f) writing data from the second processing client to the storage block;

(g) updating a version counter associated with the lock mechanism using the second processing client; and

(h) releasing the lock mechanism using the second processing client.

28. The method of claim 14, in which the step (a) of assigning includes mapping the lock mechanism to the storage block based on a state of a version counter associated with the lock mechanism.

29. The method of claim 14, in which the step (a) of assigning includes mapping the lock mechanism to the storage block based on activity of a version counter associated with the lock mechanism, and activity of at least one other version counter associated with a different lock mechanism.

30. A computer-readable medium having computer-executable instructions for performing the actions recited in claim 14.

31. The method of claim 14, in which the step (b) of accessing includes an action selected from a group consisting of:

acquiring the lock mechanism for shared use with other processing clients; and

acquiring the lock mechanism for exclusive use by the first processing client.

32. The method of claim 31, further comprising a step (c) of subsequently forcibly acquiring the lock mechanism, using the second processing client, if step (b) of accessing included acquiring the lock mechanism for exclusive use by the first processing client.

33. The method of claim 14, further comprising a step (c) of modifying an expiration timer associated with the lock mechanism.

34. The method of claim 14, further comprising a step (c) of obtaining the state of an expiration timer associated with the lock mechanism.

Description

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.

TECHNICAL FIELD OF THE INVENTION

This invention relates generally to computer systems, and particularly, but not by way of limitation, to a global file system and data storage device locks for managing shared data storage on a networked computer system.

BACKGROUND OF THE INVENTION

Modern computer systems are often linked by a network. The network couples several computer processing clients for communication using a standard communication protocol, such as Transmission Control Protocol/Internet Protocol (TCP/IP). Data stored locally at the clients is shared with other clients by sending messages across the network. Alternatively, data is stored in data storage devices that are shared by multiple clients, rather than being stored locally by the clients. Both the shared storage device and an intermediary server are coupled to the network. The server controls client access to the shared storage device. The clients send requests for data to the server. The server accesses the shared storage device and provides the requested data to the client.

Since communication across a network is typically relatively slow, data caching is often desirable to minimize communication across the network. Rather than returning the data from the client to the shared storage device after client processing, and again retrieving the data from the shared storage device for further client processing, a duplicate copy of the data is transferred from the shared storage device to the client. The copy of the data is stored locally in a memory cache in the client, where it is accessed directly. Such direct access of cached data is much faster than retrieving the data from the shared storage device over the network.

The client may perform multiple read operations on the cached data, rather than repeatedly requesting such data be delivered over the network from the shared storage device. The client may also perform operations that alter the cached data. As a result of any such alterations, the client's cached data will be different than the original data in the shared memory device. In this case, the client must eventually write back the altered cached data to the shared storage device after it completes its processing. The altered data is transferred back over the network to the shared storage device. The obsolete original data is overwritten with the altered data.

Thus, caching requires copies of data in more than one location, such as at the shared storage device and at the client. This minimizes the time spent transferring data across the network. Caching also allows data to be prefetched from the shared storage device concurrently with other client operations. Prefetching allows the data to be already resident in the client's memory cache when needed by the client. Caching ultimately requires eventual consistency between the data in client memory cache and the shared storage device.

Sharing a storage device among several clients further complicates the task of maintaining data consistency between the client memory caches and the shared storage device. In a traditional client-server system, the task of maintaining data consistency is typically performed by a centralized server. Servers are categorized as either stateless or stateful. A stateless server maintains no information regarding a client's previous requests to access data from the shared storage device. Stateless severs rely on their clients to maintain data consistency. The clients maintain data consistency by brute force techniques, such as by using write-through caching and periodically invalidating their local memory caches. A stateful server maintains information about previous client requests to access data from the shared storage device. This state information, which is typically stored in volatile memory in the server, allows the server to call back and notify the clients that: (1) their client cached data is inconsistent with the data on the shared storage device, or (2) the client cached data must be written back to the server for updating the corresponding data residing on the shared storage device.

Using the server to maintain data consistency has several drawbacks. First, a server limits how fast data can be transferred across the network when it controls access to shared storage device. The speed at which data is obtained from the shared storage device is limited by the speed of the server. Second, a server-based architecture is susceptible to server failures. Such an architecture is not robust, because a server failure prevents all clients from accessing any of the storage devices controlled by the server. Third, maintaining a separate server for controlling access to shared storage devices adds additional complexity and expense to the system.

SUMMARY OF THE INVENTION

One aspect of the present system includes a data storage device that is accessible by first and second remote processing clients. The storage device includes at least one lock. Each lock is associated with the use of at least one storage block on the storage device by each of the first and second clients. The locks control access to the storage blocks based on control signals received from a distributed file system residing on both of the first and second clients. The clients acquire the locks for exclusive use by the acquiring client, or for shared use with other clients.

In various embodiments, the locks include: a version counter that is updated when data is written to the at least one storage block controlled by the lock, an activity element that triggers updating of the version counter for both reads and writes, an expiration timer for timing out the acquisition of the lock by the client, and a world-wide names list of clients that have acquired the lock. In various embodiments, the device locks execute actions based on control signals received from the clients. These actions are selected from the group consisting essentially of Lock, Unlock, Unlock Increment, Reset Lock, Activity On, Activity Off, Lock Shared, Lock Exclusive, Force Lock Exclusive, Touch Lock, and Report Expired actions.

Another aspect of the present system includes a method of using a system. The system has first and second processing clients, a distributed file system, a data storage device shared by the first and second clients, and a network coupled to the storage device and each of the first and second clients. The data storage device includes storage blocks and a plurality of locks. The method includes assigning a lock to at least one storage block using the file system. At least one of the storage blocks is accessed, using the first client, by acquiring the lock assigned to the at least one storage block, if the lock is available. The client acquires the lock for exclusive use by the acquiring client, or for shared use with other clients.

In various further embodiments, the method includes releasing the lock after accessing the storage block and updating a version counter associated with the lock if an activity element associated with the lock is set. Data is written from the first client to the storage block. A version counter associated with the lock is updated using the first client. The first client is used to release the lock.

In one embodiment, the method includes waiting for a predetermined period of time after setting the activity element by a second client. If the version counter associated with the lock is not updated during the predetermined period of time, then the version counter is updated and the lock is released by the second client. If the version counter is updated during the predetermined period of time, then the second client clears the activity element. In one embodiment, the second client provides an input version counter value to the storage device, and if a state of the lock's version counter matches the input version counter value, then the second client updates the lock's version counter and releases the identified lock.

The lock may be acquired for shared use with other clients, for exclusive use by the acquiring client. If the lock is exclusively held by the first client, it may subsequently be forcibly acquired using the second client. The method includes resetting or otherwise modifying an expiration timer associated with the lock, and obtaining the state of an expiration timer associated with the lock.

In summary, device locks provide decentralized control of the shared data storage device on which they are located. Clients acquire the locks for excluding other clients, thereby maintaining data consistency, or for shared use by multiple clients. The device locks allow use of a serverless distributed architecture global file system (GFS). A serverless system allows higher speed data transfer across the network, eliminates the risk of server failure, and reduces system cost and complexity. Shared data storage better utilizes storage resources and simplifies redundant storage techniques. Moreover, no communication between clients is required to arbitrate for the shared resources. Direct attachment of a shared storage device to a client is susceptible to the risk of that client's failure; the present invention avoids this problem. A locally attached shared storage device also wastes a local host client's bandwidth by using it to transfer data to other clients; this is also avoided by the present invention. Moreover, data file size is not limited by the storage capacity available on any particular host client. Also, the present invention minimizes overhead steps for data transfer. These and other aspects of the invention will be apparent after reading the following detailed description of the invention and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like numerals describe substantially similar components throughout the several views.

FIG. 1 is a block diagram illustrating generally one embodiment of a distributed file system for a shared storage architecture system according to one aspect of the present invention.

FIG. 2 is a block diagram illustrating generally one embodiment of the present invention using a fibre channel interface.

FIG. 3 is a block diagram illustrating conceptually several network interconnection configurations supported by the fibre channel interface.

FIG. 4 is a block diagram illustrating conceptually one embodiment of a GFS architecture according to one aspect of the present invention.

FIG. 5 is a conceptual diagram illustrating generally one embodiment of a GFS file system mapping structure according to one aspect of the present invention.

FIG. 6 illustrates generally one embodiment of certain GFS file system constructs.

FIG. 7 is a block diagram illustrating generally one embodiment of a storage device in which locks are provided for controlling access to data storage blocks, such as for maintaining data consistency.

FIG. 8A is a block diagram illustrating generally one configuration of an array of locks.

FIG. 8B is a block diagram illustrating generally another configuration of an array of locks.

FIG. 9 is a table illustrating generally one possible sequence of events undertaken by a first client and a second client in accessing shared data.

FIG. 10 is a table illustrating generally another possible sequence of events undertaken by a first client and a second client in accessing shared data.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that the embodiments may be combined, or that other embodiments may be utilized and that structural, logical and electrical changes may be made without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims and their equivalents.

In this document, the terms "client" or "clients" refers to a processing client, for example a microprocessor-based computer system, or any other device capable of communicating electromagnetic or other data signals.

In this document, the term "storage device" refers to any device or media capable of storing data, together with any peripheral circuits, including those associated with performing the function of storing, retrieving, or otherwise communicating data. The storage device can include either nonvolatile or volatile storage media including, but not limited to: electronic storage media such as dynamic random access memories (DRAMs), flash electrically erasable and programmable read-only memories (EEPROMs), magnetic tape or disk storage media (e.g., hard drive), electromagnetic or optical storage media, or any other analog, digital, or other data storage media or array of media.

In this document, the terms "node" or "nodes" refers broadly to any client, storage device, or other device that is communicatively coupled to a network or other communication interface.

In this document, the term "network" refers to any synchronous or asynchronous data transfer medium or other communication device, and can include any other "networks" to which a first "network" can be communicatively coupled. The network can provide wired or wireless communication, including, but not limited to: electrical coupling for communication of electrical signals, electromagnetic coupling for communication of electromagnetic/optical signals, or using any other communicative coupling technique. In particular, the term "network" includes a "fibre channel interface," described below.

In this document, the term "computer readable medium" includes any storage device or network, as defined above, or any electronic device capable providing data, instructions, or commands.

In this document, the term "storage block" refers broadly to any unit of memory of any size or storage capacity. Multiple storage blocks may also be referred to collectively as a "storage block."

Overview of Global File System (GFS) for Shared Storage

FIG. 1 is a block diagram illustrating generally, by way of example, but not by way of limitation, one embodiment of a distributed file system, referred to as a Global File System (GFS), for a shared storage architecture system 100 according to one aspect of the present invention. System 100 includes a plurality of computer or simply as clients 105. Each client 105 is communicatively coupled to a network 110. System 100 also includes at least one data storage device 115 shared by at least some of the clients 105. FIG. 1 illustrates, by way of example, but not by way of limitation, a plurality of storage devices 115A, . . . ,115N, such as disk media. Each is referred to generally as storage device 115. Each storage device 115 includes a communication interface that is communicatively coupled to the clients 105 through the network 110. Each storage device 115 includes, for example, data storage media and peripheral circuits. According to one aspect of the invention, data files stored on the storage media of one of the storage devices 115 are capable of being accessed by a plurality of the clients 105.

Characteristics of Shared Storage Architectures

According to another aspect of the invention, system 100 provides a distributed file system in which the clients 105 can access data stored on the shared storage devices 115. Such a shared storage architecture has several advantages over a message-based architecture in which clients 105 share only data stored on other clients 105. According to one aspect of the invention, the shared storage approach advantageously allows every client 105 uniform access to all storage devices 115. Also, individual clients 105 are not required to service requests for data from other clients 105, thereby freeing individual clients 105 for performing other tasks. Moreover, clients 105 are relatively unaffected by the failure of other clients 105 (i.e., a client 105 can still obtain data from a storage device 115 even if operation of other clients 105 is halted or suspended).

In one embodiment, accessibility and robustness of the shared storage devices 115 is improved by using a redundant array of independent and/or inexpensive disks (RAID) configuration of storage devices 115. A RAID configuration duplicates data on different ones of storage devices 115 for simultaneous access by different clients 105, thereby increasing the reliability of system 100. A RAID configuration provides redundancy in the event of a failure by one of storage devices 115, since redundant data is stored on others of the storage devices 115. In one embodiment of the present invention, storage devices 115 include command queuing for optimizing head seeks of a magnetic or other storage media, and further improving data throughput.

cl Eliminating the Server in a Shared Storage System

According to another aspect of the invention, system 100 is serverless. Clients 105 can access storage devices 115 directly, rather than through a server. In this respect, system 100 of the present invention is different from a traditional clientserver system, in which clients send requests to a server, which in turn provides data or metadata (i.e., data that describes other data) in response to the client requests. Servers can be implemented as general-purpose computers that can also execute other computing or graphics applications. Servers can also be implemented as dedicated computers performing only the specialized task of file serving, using a conventional standard such as Sun Microsystems'Network File System (NFS) and a protocol such as TCP/IP for communicating the data in messages across the network. System 100 of the present invention, however, eliminates the need for expensive server hardware and allows for faster communication of the data across network 110.

Characteristics of Message-Based Architectures

In a message-based system, several client computing nodes are intercoupled for communicating to each other via the network using a standard communication protocol, such as TCP/IP. Data stored locally at the clients can be shared with other clients by sending the data within messages carried across the network. Advantages of a message-based architecture include its extensibility and its portability to many platforms (e.g., Sun Microsystems, Inc.'s Network File System (NFS), or the Coda file system developed by Carnegie Mellon University). The high level message-based communication protocol allows a potentially large number of clients to be interconnected to the network.

While message-based systems perform well when data access is well balanced across the various clients, such load balancing is difficult to achieve because processing capability and workload typically varies between clients. Localization of data is difficult to maintain because the storage resources of particular ones of the clients will be more highly demanded than the storage resources of other ones of the clients. Using multiple copies of the data will decrease such problems, but at the increased cost of maintaining a data coherence mechanism. Furthermore, message-based systems obtain less benefit from high speed disk arrays, since the bandwidth of each client and storage device is limited to the bandwidth of the message-based network.

Message-based systems are also prone to failure of the file server or failure of the client-based storage devices. A server failure may render data inaccessible to all of the various clients. Redundant disk arrays at each client provide tolerance to failures of individual disks, but they do not protect against failures of the clients. Increased fault tolerance requires that software redundancy schemes be built into the file system.

Fibre Channel

Networks are characterized by how they connect clients, shared storage devices, or other nodes. For example, networks are characterized as shared media networks and point-to-point networks. Shared media networks (e.g., Ethernet) communicatively couple clients together via a common bus or ring. This allows only two nodes to communicate at a particular time. For example, in a token-ring configuration, various nodes may arbitrate for the ring by acquiring a token that excludes other nodes from using the ring. Alternatively, the nodes may simply send data over the ring without first acquiring a token or undertaking any other form of arbitration for the ring. If a collision of data is detected, the data is sent again. By contrast, point-to-point networks include switched networks having parallel interconnection paths between nodes. Multiple communication channels between separate nodes may operate simultaneously. This allows communication between more than two nodes at a particular time. Examples of communication over switched networks include Asynchronous Transfer Mode (ATM) and High Performance Parallel Interface (HIPPI).

FIG. 2 is a block diagram illustrating generally, by way of example, but not by way of limitation, one embodiment of the present invention. This embodiment uses a particular kind of network 110, referred to as a fibre channel interface 200. One example of a serial fibre channel interface 200 is described in the American National Standards Institute (ANSI) X3T9.3 standard, available from ANSI, New York, N.Y., the disclosure of which is incorporated herein by reference. Fibre channel interface 200 combines the features of network (e.g., ATM or Fiber Distributed Data Interface (FDDI)) and storage interfaces (e.g., parallel Small Computer System Interface (SCSI) and Intelligent Peripheral Interface (IPI)). Fibre channel interface 200 also combines the extensibility of a message-based architecture with numerous advantages afforded by a shared storage device architecture.

FIG. 3 is a block diagram illustrating conceptually, by way of example, but not by way of limitation, several network interconnection configurations supported by fibre channel interface 200. Fibre channel interface 200 supports both ring and switched network interconnection topologies and provides scalable network-attached storage. In FIG. 3, a plurality of ports or nodes 300 are communicatively intercoupled, such as in a point-to-point configuration 305, an arbitrated loop configuration 310, and a ring configuration 315. Moreover, these configurations may themselves be interconnected, for example, point-to-point configuration 305 is illustrated as being interconnected to arbitrated loop configuration 310. In this way, fibre channel interface 200 allows a plurality of parallel connections to be aggregated through a shared port or node 300 with another plurality of similarly or differently configured connections, thereby offering nearly unlimited scalability and extensibility.

In one embodiment, by way of example, but not by way of limitation, fibre channel interface 200 includes up to 126 nodes that are connected in a single loop, as illustrated conceptually by arbitrated loop configuration 310. By comparison, a SCSI interface typically only allows a more limited number of nodes (e.g., 7 nodes for SCSI-1 and 16 nodes for SCSI-2) to be daisy-chained in an arbitrated loop configuration. Thus, the scalability of fibre-channel interface 200 typically significantly exceeds that of a parallel SCSI interface. Moreover, one embodiment of fibre channel interface 200 allows peak transfer rates that exceed 100 megabytes per second. Other planned embodiments of fibre-channel interface 200, include peak transfer rates of 400 megabytes per second. As set forth above, fibre channel interface 200 provides high speed data transfer as well as scalability with respect to both bandwidth (e.g., the peak transfer rate) and the number of devices that can be interconnected. In one embodiment, by way of example, but not by way of limitation, various nodes 300 are linked to each other by fiber optics having link lengths of up to 10 kilometers. In another embodiment, by way of example, but not by way of limitation, various nodes 300 are linked to each other by copper wire having link lengths of up to 25 meters.

GFS Example Embodiment

FIG. 4 is a block diagram illustrating conceptually, by way of example, but not by way of limitation, one embodiment of a system 100 using one embodiment of a GFS architecture. In FIG. 4, system 100 includes a plurality of clients 105A-N, communicatively intercoupled by a network 110, referred to as a Storage Area Network (SAN). One embodiment of network 110 includes fibre channel interface 200. System 100 also includes a Network Storage Pool (NSP) 400.

Network Storage Pool 400 includes one or more network attached storage devices that are capable of being shared by a plurality of computer processing or other clients 105. According to one aspect of the invention, Network Storage Pool 400 includes a collection of network attached storage devices that are logically grouped to provide clients 105 with a unified shared storage space. In one embodiment, Network Storage Pool 400 is not owned or controlled by any one of clients 105. Instead, this embodiment allows Network Storage Pool 400 to provide shared storage that is capable of being used by all clients 105 or other nodes attached to network 110.

In the embodiment illustrated in FIG. 4, Network Storage Pool 400 includes at least one subpool, such as the plurality of subpools 400A-N illustrated by way of example in FIG. 4. According to one aspect of the invention, each of subpools 400A-N inherits attributes characteristic of the underlying hardware or storage media providing data storage. In one embodiment, for example, subpool 400A is a solid state device providing volatile or nonvolatile storage, such as a static random access memory (SRAM), dynamic random access memory (DRAM), flash electrically erasable and programmable read-only memory (EEPROM), or other integrated circuit memory device. In another example, subpool 400B includes a magnetic or optical disk, tape, or other such storage device. In another example, subpool 400C is a disk array in a RAID-5 configuration (e.g., data is striped block-by-block across multiple disks, and parity data is also spread out over multiple disks). In another example, subpool 400D includes an array of software striped disks. Striping spreads sequentially accessed data across many disks so that multiple disks can be accessed in parallel to increase performance. Striping is typically performed by a hardware RAID controller. However, striping can also be performed by the client computer, which is referred to as software striping. In a further example, subpool 400N is a disk array in a RAID-3 configuration (e.g., data is striped byte-by-byte across multiple disks and a single additional disk is dedicated to recording parity data). Other RAID configurations or other redundant data storage schemes can also be used.

In FIG. 4, Network Storage Pool 400 is illustrated conceptually as providing shared network attached storage that is capable of being used by a plurality of the clients 105. The clients 105 are allowed direct channel data transfer with the storage devices 115 in the subpools of Network Storage Pool 400. This approach should be contrasted with message-based distributed file systems. Message-based distributed file systems require a server that acts as an intermediary, thereby masking some of the attributes of the storage devices 115 (e.g., not fully utilizing the speed of a solid state storage device as illustrated by subpool 400A).

In one embodiment, Network Storage Pool 400 comprises network attached storage devices 115 that are physically separate from ones of the clients 105, as illustrated in FIG. 2. In another embodiment, Network Storage Pool 400 includes some storage devices 115 that are physically located together with ones of the clients 105, but accessible for use by others of the clients 105. In a further embodiment, Network Storage Pool 400 includes some storage devices 115 that are physically located together with ones of the clients 105, and other storage devices 115 that are physically located elsewhere. All of the storage devices 115 or subpools 400A-N are capable of being accessed by a plurality of the clients 105.

This embodiment of system 100 advantageously provides large data storage capacity and high bandwidth for data transfer, such as for multimedia, scientific computing, visualization, and other suitable applications. In one example, system 100 includes the GFS distributed file system, and is implemented in the IRIX operating system by Silicon Graphics, Inc. (SGI) of Mountain View, Calif., under the Virtual File System (VFS) interface. VFS permits an operating system to simultaneously support multiple file systems. In this embodiment, the GFS is accessed using standard Unix commands and utilities.

In one embodiment of system 100, data is cached in the main memories of the computer processing or other clients 105 only during input/output (I/O) request processing. For example, after each data request by a client 105, a copy of the data is transferred from the Network Storage Pool 400 to the client 105. After the client 105 reads the data, the data is released (e.g., other clients 105 are allowed to access the storage blocks in the Network Storage Pool 400 that contain the data). If the client 105 has modified the data, the modified data is written back from the main memory of the client 105 to the Network Storage Pool 400 (e.g., after which other clients 105 are allowed to access the storage blocks in the Network Storage Pool 400 that contain the data).

According to another aspect of the invention, for example, GFS caches data to exploit locality of reference on the storage devices 115 comprising Network Storage Pool 400, such that successive I/O requests access data that is clustered together in storage. GFS informs the clients 105 on each I/O request of what data is appropriate to cache, such as metadata and frequently-accessed small files (e.g., directories). In one embodiment, consistency between the copy of the data cached at the client 105 and the original data maintained in the Network Storage Pool 400 is maintained by data storage device locks. The device locks facilitate atomic read, modify, and write operations, and are implemented by a controller, microprocessor, or other peripheral circuit included within ones of the storage devices 115 forming the subpools of Network Storage Pool 400. Each device lock gives a particular client 105 exclusive access to one or more storage blocks residing on the storage media of a storage device 115 and controlled by the device lock. Other clients 105 are usually excluded from accessing the locked one or more storage blocks until the particular client releases the lock. According to one aspect of the present invention, the device locks provide the an easy-to-use decentralized data consistency mechanism. According to another aspect of the invention, the device locks provide a robust data consistency mechanism. Because the locks are distributed across a large number of storage devices 115, they are less susceptible to failure. According to a further aspect of the invention, no messages need be passed between clients 105 in order to maintain data consistency.

Some GFS Advantages

According to one aspect of the invention, system 100 includes a distributed architecture file system (e.g., GFS) that eliminates the master/slave architecture found in most present distributed client/server computing environments. In one embodiment, system 100 includes a plurality of clients 105 that access storage devices 115 through a network 110. One example of a fast switched network 110 includes fibre channel interface 200, providing low time latency for accessing data and high bandwidth for communicating data. Other examples of network 110 include, but are not limited to: a fast bus, multistage network, or crossbar switch. The present invention includes many advantages, such as described below.

For example, one aspect of the invention provides a storage architecture that allows a storage system designer and administrator to pool disk drives and other storage devices into a shared disk memory in Network Storage Pool 400 or other shared data storage arrangement. This provides better utilization of storage resources. This also accommodates redundancy (e.g., RAID) across various ones of the storage devices in Network Storage Pool 400. Moreover, building redundancy into a disk array, according to the present invention, is easier to accomplish than existing techniques for insuring that a complicated server (including associated hardware, software, and network connections) does not fail.

According to another aspect of the invention, Network Storage Pool 400 provides shared storage resources that are substantially equally accessible to each of clients 105. The present invention obtains this advantage without using a server or other intermediation. Neither does the present invention require communication between clients 105 while arbitrating for use of the shared storage resources in Network Storage Pool 400.

Another aspect of the invention provides a more robust system by eliminating the problem of having a single point-of-failure that exists when a shared storage device is attached to a single host client 105. In such a case, if the host client fails, the shared storage device becomes unavailable to the other clients 105. By contrast, in the present system, if a client 105 fails, the shared storage devices 115 in Network Storage Pool 400 are still available for other clients 105 to access.

Another aspect of the invention exploits bandwidth capabilities both within and across next-generation personal computers (PCS), desktop workstations, high-performance servers, and supercomputers. For example, the present invention does not require that the bandwidth of a local host client 105 be wasted in transferring data from a locally attached storage device to another host client 105.

According to another aspect of the invention, the size of the distributed architecture file system, GFS, and consequently, the size of a single file, is not limited by the storage capacity available on any particular host client. In one embodiment, GFS spans multiple storage devices 115.

According to a further aspect of the invention, each client 105 that is coupled to the network 110 effectively views the network attached storage devices 115 as being locally attached. Thus, GFS provides a more direct connection between client 105 and storage device 115 by bypassing protocol stack overheads. Table 1 illustrates exemplary steps required for accessing a file using a file system such as NFS. Table 2 lists exemplary steps required for accessing a file using a file system such as GFS. As seen by comparing Tables 1 and 2, the GFS file system of the present invention requires less overhead steps than NFS.

                             TABLE 1
               Steps For Accessing Files Using NFS
              Step                     Where performed
              NFS                           Client
              XDR/RPC                       Client
              TCP/IP                        Client
              Ethernet                     Network
              TCP/IP                        Server
              NFS                           Server
              VFS                           Server
              SCSI Driver                   Server
              SCSI Connection               Server
              Disk Access                   Server


                         TABLE 2
               Steps For Accessing Files Using GFS
            Step                       Where Performed
            GFS                             Client
            SCSI Driver                     Client
            SCSI on Fibre Channel              Client
            Disk Access                 Client/Network

                             TABLE 3
           Example 1 of One Embodiment Of DLock Actions
    Action                 Description of Action
    Lock                   Test-and-Set Action
                           if (Lock[ID].state = 1)
                            then
                             Return.result .rarw. 0
                            else
                             Return.result .rarw. 1
                             Lock[ID].state .rarw. 1
    Unlock                 Clear Action
                           Return.result .rarw. 1
                           Lock[ID].state .rarw. 0
                           if (Lock[ID].activity = 1)
                            then
                             Increment Lock[ID].version
    Unlock Increment       Clear Action
                           Return.result .rarw. 1
                           Lock[ID].state .rarw. 0
                           Increment Lock[ID].version
    Reset Lock             Conditional Clear Action
                           if (Lock[ID].version = (input version value))
                            then
                             Return.result .rarw. 1
                             Lock[ID].state .rarw. 0
                             Increment Lock[ID].version
                            else
                             Return.result .rarw. 0
    Activity On            Turn On Activity Monitor
                           Lock[ID].activity .rarw. 1
                           Return.result .rarw. 1
    Activity Off           Turn Off Activity Monitor
                           Lock[ID].activity .rarw. 0
                           Increment Lock[ID].version
                           Return.result .rarw. 1
    No Operation           No Operation
                           Return.result .rarw. 1
    Included with each of  After Each Action
    Lock, Unlock, Unlock   Return.state .rarw. Lock[ID].state
    Increment, Reset Lock, Return.activity .rarw. Lock[ID].activity
    Activity On, Activity Off, Return.version .rarw. Lock[ID].version
    and No Operation actions

                             TABLE 4
                   Codes Defining SCSI Commands
              Function           Hexadecimal Code Value
              DLock                        A0
              Mode Page                    29


                         TABLE 5
                    DLock Command of Example 1
     Byte   Bit 7 Bit 6 Bit 5  Bit 4  Bit 3  Bit 2  Bit 1  Bit 0
       0          DLock Command = A0 (hexadecimal value)
       1            Reserved            Action (See Table 6)
       2                 Lock Identification (ID)
       3                         (4 bytes)
       4
       5
       6                    Input Version Value
       7                         (4 bytes)
       8
       9
      10                     Allocation Length
      11                       Control Byte

                             TABLE 6
    Action Codes for defining a particular DLock command of Example 1
                Action            Hexadecimal Code Value
             No Operation                    0
                 Lock                        1
                Unlock                       2
            Unlock Increment                  3
              Reset Lock                     4
              Activity On                    5
             Activity Off                    6
               Reserved                 7 through F

                             TABLE 7
                          Returned Data
    Byte   Bit 7  Bit 6  Bit 5  Bit 4  Bit 3  Bit 2  Bit 1  Bit 0
      0   Result  State Activ-              Reserved
                          ity
      1                  Returned Version Value
      2                        (4 bytes)
      3
      4
      5                         Reserved
      6                         Reserved
      7                         Reserved

                             TABLE 8
        SCSI Device Locks mode page (Hexadecimal values).
    Byte   Bit 7  Bit 6  Bit 5  Bit 4  Bit 3  Bit 2  Bit 1  Bit 0
      0     PS             Vendor Unique Page Code=29
      1                     Page Length =06
      2                   Lock Size (Encoded)
      3              Supported Lock Sizes (Encoded)
      4                Number of Supported Locks
      5                        (4 bytes)
      6
      7

                             TABLE 9
     Example of Encoding Lock Size in Device Locks mode page
                                            Version
                               Activity     Counter
      Binary    State Element Element 800 Element 810 Total Size of
     Encoding     805 Size       Size        Size      Lock 715
     00000001       1 bit        1 bit      22 bits     24 bits
     00000010       1 bit        1 bit      30 bits     32 bits


                         TABLE 10
    Example of Encoding Supported Lock Size in Device Locks mode page
                                            Version
                               Activity     Counter
      Binary    State Element Element 800 Element 810 Total Size of
     Encoding     805 Size       Size        Size      Lock 715
     xxxxxxx1       1 bit        1 bit      22 bits     24 bits
     xxxxxx1x       1 bit        1 bit      30 bits     32 bits

                             TABLE 11
           Example 2 of One Embodiment of DLock Actions
    Action            Description of Action
    Lock Shared       if (Lock[ID].state = U = 0 hex) then
                       If Lock[ID].expired = ExpiredFromLock
                         Exclusive
                        then
                         Lock[ID].state .rarw. (E = 2 hex)
                        else
                         Lock[ID].state .rarw. (S = 1 hex);
                       Return.result .rarw. 1
                       Lock[ID].holders .rarw. 1
                       Add WWN to list
                       Reset expiration timer;
                      if (Lock[ID].state = S = 1 hex) then
                       if (Lock[ID].holders < MaxHolders) then
                        Return.result .rarw. 1
                        Increment Lock[ID].holders
                        Add WWN to list
                        Reset Expiration Timer;
                       else
                        Return.result .rarw. 0;
                      if (Lock[ID].state = E = 2 hex) then
                       if Lock[ID].wwn[0] = wwn then
                        Return.result .rarw. 1
                        Lock[ID].state .rarw. S = 1 hex
                        Reset expiration timer
                       else
                        Return.result .rarw. 0;
    Lock Exclusive    if (Lock[ID].state = U = 0 hex) then
                       Lock[ID].state .rarw. (E = 2 hex)
                       Return.result .rarw. 1
                       Lock[ID].holders .rarw. 1
                       Add WWN to list
                       Reset expiration timer;
                      if (Lock[ID].state = S = 1 hex) then
                       if (Lock[ID].holders = 1 and Lock[ID].wwn[0]
                         =wwn