Redundant storage for multiple processors in a ring network

Abstract

A dual-axis RAID system includes a plurality of X-axis ordinal series of disks, configured to store parity data and a tape drive, and a Y-axis ordinal series of parity disks. The Y-axis series is smaller than the X-axis series, because the X-axis series contains an extra disk configured as a segment journal disk. The RAID system communicates with clients on a network a network via an SCI network interface.

Claims

What is claimed is:

1. A computerized system for providing high speed fault-tolerant storage of information, the system comprising:

an array of computer storage devices, said computer storage devices being capable of storing computer-readable information,

said array of computer storage devices including a plurality of computer storage devices,

computer-readable information stored on said plurality of computer storage devices,

said computer-readable information being stored on said plurality of computer storage devices in data redundant fashion,

said array being configured to appear as a unitary block of computer-readable storage to applications programs,

an adapter for accessing said array,

said adapter including an array interface for performing desired operations on said array, such as read and write operations,

said adapter including a network interface for interfacing with a computer network in order to permit users on a computer network to access said array,

said network interface operating according to a scalable coherent interface (SCI) protocol,

a computer network,

said computer network being arranged in ring topology,

said computer network utilizing a unidirectional interfaces in order to minimize latency,

a plurality of client processors connectable to said computer network,

said computer network supporting distributed processing of tasks across a plurality of processors connected to said computer network,

a block of memory shared by a plurality of processors on said network; and

wherein said array selects at least one feature selected from the group consisting of on-demand storage journaling capability, hotfix redirection, mirrored caching, annotated storage journaling, dynamic stripe block allocation, dynamically added stripe and mirror sets, break-away mirroring, and infinite HSM storage journaling.

2. A system as recited in claim 1 therein said computer network employs dual ring topology in order to permit continued operation of said computer network in the event of failure of a node of said computer network.

3. A system as recited in claim 1 wherein said array includes dual data parity information for use in data error correction.

4. A system as recited in claim 1 wherein said computer network includes a switch to provide continued operation of said network in case of failure of a node in said network.

5. A system as recited in claim 1 wherein said SCI interface incorporates a plurality of axes in said array forming a fault-tolerant interconnection fabric.

6. A computerized system for providing high speed fault-tolerant storage of information, the system comprising:

an array of computer storage devices, said computer storage devices being capable of storing computer-readable information,

said array of computer storage devices including a plurality of computer storage devices,

computer-readable information stored on said plurality of computer storage devices,

said computer-readable information being stored on said plurality of computer storage devices in data redundant fashion,

said array being configured to appear as a unitary block of computer-readable storage to applications programs,

an adapter for accessing said array,

said adapter including an array interface for performing desired operations on said array, such as read and write operations,

said adapter including a network interface for interfacing with a computer network in order to permit users on a computer network to access said array,

said network interface operating according to a scalable coherent interface (SCI) protocol,

a computer network,

said computer network being arranged in ring topology,

said computer network utilizing a unidirectional interfaces in order to minimize latency,

a plurality of client processors connectable to said computer network,

said computer network supporting distributed processing of tasks across a plurality of processors connected to said computer network,

a block of memory shared by a plurality of processors on said network; and

wherein said array selects a plurality of features selected from the group consisting of on-demand storage journaling capability, hotfix redirection, minored caching, annotated storage journaling, dynamic stripe block allocation, dynamically added stripe and mirror sets, break-away mirroring, and infinite HSM storage journaling.

7. A system as recited in claim 6 wherein said computer network employs dual ring topology in order to permit continued operation of said computer network in the event of failure of a node of said computer network.

8. A system as recited in claim 6 wherein said array includes dual data parity information for use in data error correction.

9. A system as recited in claim 6 wherein said computer network includes a switch to provide continued operation of said network in case of failure of a node in said network.

10. A system as recited in claim 6 wherein said SCI interface incorporates a plurality of axes in said array forming a fault-tolerant interconnection fabric.

11. A computerized system for providing high speed fault-tolerant storage of information, the system comprising:

an array of computer storage devices, said computer storage devices being capable of storing computer-readable information,

said array of computer storage devices including a plurality of computer storage devices,

computer-readable information stored on said plurality of computer storage devices,

said computer-readable information being stored on said plurality of computer storage devices in data redundant fashion,

said array being configured to appear as a unitary block of computer-readable storage to applications programs,

an adapter for accessing said array,

said adapter including an array interface for performing desired operations on said array, such as read and write operations,

said adapter including a network interface for interfacing with a computer network in order to permit users on a computer network to access said array,

said network interface operating according to a scalable coherent interface (SCI) protocol,

a computer network,

said computer network being arranged in ring topology,

said computer network utilizing a unidirectional interfaces in order to minimize latency,

a plurality of client processors connectable to said computer network,

said computer network supporting distributed processing of tasks across a plurality of processors connected to said computer network,

a block of memory shared by a plurality of processors on said network; and

wherein said array selects at least one feature selected from the group consisting of on-demand storage journaling capability, hotfix redirection, mirrored caching, and annotated storage journaling.

12. A system as recited in claim 11 wherein said computer network employs dual ring topology in order to permit continued operation of said computer network in the event of failure of a node of said computer network.

13. A system as recited in claim 11 wherein said array includes dual data parity information for use in data error correction.

14. A system as recited in claim 11 wherein said computer network includes a switch to provide continued operation of said network in case of failure of a node in said network.

15. A system as recited in claim 11 wherein said SCI interface incorporates a plurality of axes in said array forming a fault-tolerant interconnection fabric.

16. A computerized system for providing high speed fault-tolerant storage of information, the system comprising:

an array of computer storage devices, said computer storage devices being capable of storing computer-readable information,

said array of computer storage devices including a plurality of computer storage devices,

computer-readable information stored on said plurality of computer storage devices,

said computer-readable information being stored on said plurality of computer storage devices in data redundant fashion,

said array being configured to appear as a unitary block of computer-readable storage to applications programs,

an adapter for accessing said array,

said adapter including an array interface for performing desired operations on said array, such as read and write operations,

said adapter including a network interface for interfacing with a computer network in order to permit users on a computer network to access said array,

said network interface operating according to a scalable coherent interface (SCI) protocol,

a computer network,

said computer network being arranged in ring topology,

said computer network utilizing a unidirectional interfaces in order to minimize latency,

a plurality of client processors connectable to said computer network,

said computer network supporting distributed processing of tasks across a plurality of processors connected to said computer network,

a block of memory shared by a plurality of processors on said network; and

wherein said array selects at least one feature selected from the group consisting of dynamic stripe block allocation, dynamically added stripe and mirror sets, break-away mirroring, and infinite HSM storage journaling.

17. A system as recited in claim 16 wherein said computer network employs dual ring topology in order to permit continued operation of said computer network in the event of failure of a node of said computer network.

18. A system as recited in claim 16 wherein said array includes dual data parity information for use in data error correction.

19. A system as recited in claim 16 wherein said computer network includes a switch to provide continued operation of said network in case of failure of a node in said network.

20. A system as recited in claim 16 wherein said SCI interface incorporates a plurality of axes in said array forming a fault-tolerant interconnection fabric.

21. A computerized system for providing high speed fault-tolerant storage of information, the system comprising:

an array of computer storage devices, said computer storage devices being capable of storing computer-readable information,

said array of computer storage devices including a plurality of computer storage devices,

computer-readable information stored on said plurality of computer storage devices,

said computer-readable information being stored on said plurality of computer storage devices in data redundant fashion,

said array being configured to appear as a unitary block of computer-readable storage to applications programs,

an adapter for accessing said array,

said adapter including an array interface for performing desired operations on said array, such as read and write operations,

said adapter including a network interface for interfacing with a computer network in order to permit users on a computer network to access said array,

said network interface operating according to a scalable coherent interface (SCI) protocol,

a computer network,

said computer network being arranged in ring topology,

said computer network utilizing a unidirectional interfaces in order to minimize latency,

a plurality of client processors connectable to said computer network,

said computer network supporting distributed processing of tasks across a plurality of processors connected to said computer network,

a block of memory shared by a plurality of processors on said network; and

wherein said array selects at least one feature selected from the group consisting of on-demand storage journaling capability, mirrored caching, dynamic stripe block allocation, and break-away mirroring.

22. A system as recited in claim 21 wherein said computer network employs dual ring topology in order to permit continued operation of said computer network in the event of failure of a node of said computer network.

23. A system as recited in claim 21 wherein said array includes dual data parity information for use in data error correction.

24. A system as recited in claim 21 wherein said computer network includes a switch to provide continued operation of said network in case of failure of a node in said network.

25. A system as recited in claim 21 wherein said SCI interface incorporates a plurality of axes in said array forming a fault-tolerant interconnection fabric.

26. A computerized system for providing high speed fault-tolerant storage of information, the system comprising:

an array of computer storage devices, said computer storage devices being capable of storing computer-readable information,

said array of computer storage devices including a plurality of computer storage devices,

computer-readable information stored on said plurality of computer storage devices,

said computer-readable information being stored on said plurality of computer storage devices in data redundant fashion,

said array being configured to appear as a unitary block of computer-readable storage to applications programs,

an adapter for accessing said array,

said adapter including an array interface for performing desired operations on said array, such as read and write operations,

said adapter including a network interface for interfacing with a computer network in order to permit users on a computer network to access said array,

said network interface operating according to a scalable coherent interface (SCI) protocol,

a computer network,

said computer network being arranged in ring topology,

said computer network utilizing a unidirectional interfaces in order to minimize latency,

a plurality of client processors connectable to said computer network,

said computer network supporting distributed processing of tasks across a plurality of processors connected to said computer network,

a block of memory shared by a plurality of processors on said network; and

wherein said array selects at least one feature selected from the group consisting of hotfix redirection, annotated storage journaling, dynamically added stripe and mirror sets, and infinite HSM storage journaling.

27. A system as recited in claim 16 wherein said computer network employs dual ring topology in order to permit continued operation of said computer network in the event of failure of a node of said computer network.

28. A system as recited in claim 17 wherein said array includes dual data parity information for use in data error correction.

29. A system as recited in claim 18 wherein said computer network includes a switch to provide continued operation of said network in case of failure of a node in said network.

30. A system as recited in claim 19 wherein said SCI interface incorporates a plurality of axes in said array forming a fault-tolerant interconnection fabric.

Description

BACKGROUND OF THE INVENTION

In 1992, the Institute for Electronics and Electrical Engineers (IEEE) established a standard for a scalable coherent interface {SCI}. SCI supports distributed multiprocessing at high bandwidth, with low latency, and providing scalable architecture. Since the 1980's, there has been continuous development of redundant arrays of inexpensive or independent drives (RAID) which provide fault-tolerant data storage and access capabilities. It is against the general background of RAID and SCI that the invention is made.

BRIEF SUMMARY OF THE INVENTIONS

The invention relates generally to high-speed fault tolerant storage systems. More particularly, the invention relates to fault tolerant storage systems utilizing RAID and/or SCI, combinations thereof, and features usable with one of both of them, or otherwise usable in a data storage or data access environment. Detailed information on various example embodiments of the invention are provided in the Detailed Description below, and the invention is defined by the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A depicts an SCI system arranged in a ring topology interconnection.

FIG. 1B depicts an SCI system arranged in a dual-ring topology with data flow in two directions.

FIG. 1C depicts an SCI system interconnected by an SCI switch.

FIG. 1D depicts a dual-axis, ring based SCI interconnection network.

FIG. 2A depicts a two drive RAID 0 configuration.

FIG. 2B depicts a two drive RAID 1 configuration.

FIG. 2C depicts a seven drive RAID 2 configuration.

FIG. 2D depicts a three drive RAID 4 configuration.

FIG. 2E depicts a three drive RAID 5 configuration.

FIG. 2F depicts a four drive RAID 6 configuration.

FIG. 2G depicts a four drive RAID 1+0 configuration.

FIG. 3A depicts a hardware RAID controller connected to a processor and two disk drives.

FIG. 3B depicts a software RAID controller on a processor connected to two disk drives.

FIG. 4A depicts a RAID system having redundancy in two planes.

FIG. 4B depicts a RAID system having redundancy in two planes and facilities for HSM segment journaling and tape backup.

FIG. 4C depicts a dual-planar parity RAID system being configured to provide two RAID 1 HSM systems.

FIG. 4D depicts a dual-planar parity RAID system being configured to provide one RAID 1 HSM system with three virtual mirrored drives.

FIG. 5 illustrates the advantages of a dual-planar parity RAID system with respect to reliability and recovery of data in the event of drive failure.

FIG. 6 illustrates the advantages of a dual-planar parity RAID system configured to provide two RAID 1 virtual drives with respect to reliability against drive failure.

FIG. 7 depicts a simple RAID network subsystem that may provide RAID storage to clients.

FIG. 8 depicts the components of one implementation of a RAID network subsystem.

FIGS. 9A and 9B depict the front and rear views of one implementation of a RAID network subsystem expansion cabinet.

FIG. 10 depicts one interconnection configuration of RNS expansion cabinets through an SCI switch to four clients using several protocols.

FIG. 11 depicts one interconnection configuration of RNS expansion cabinets through an Ethernet network to four clients using several protocols.

FIG. 12 depicts the logical connections of four clients to an RNS expansion cabinet through several protocols.

FIG. 13 depicts a configuration providing a DASD string using RAID 5+1 with failover and breakaway mirror capabilities.

FIG. 14 depicts several Linux client configurations of connections to RNS devices of several configurations.

FIG. 15 depicts several Windows 2000 client configurations of connections to RNS devices of several configurations.

FIG. 16 depicts the several parts of a hot-fixable disk partition or segment.

FIG. 17 depicts the configuration of RAID segments within an M2CS partition.

FIG. 18 depicts a logical combination of two M2CS partitions to form two virtual disks.

FIG. 19 depicts the hotfix table area of a segment or partition.

FIG. 20 depicts a hotfix recovery method using a RAID 1+0 system.

FIG. 21 depicts a hotfix recovery method using a RAID 5 system.

FIG. 22 depicts a method of distributing read operations to a mirrored RAID array.

FIG. 23 depicts a method of re-mirroring a RAID 1 array for inclusion.

FIG. 24 depicts snapshots of a journalled disk.

FIG. 25 depicts a method of recording disk snapshots to a journal of journal segments.

FIG. 26 depicts real time infinite segment journaling on a dual-disk system.

FIG. 27 depicts disaster recovery using an infinite HSM segmented journal.

FIG. 28 depicts the use of a checkpoint bitmap to determine the finality of a disaster recovery using an infinite HSM segmented journal.

FIG. 29A illustrates a real time journaling method.

FIG. 29B illustrates a snapshot journaling method.

FIG. 30 depicts the operation of a real time infinite segment journaling system having two virtual RAID disks.

FIG. 31 depicts a tape master catalog with respect to tape segments.

FIG. 32 depicts an RNS system that includes the Linux operating system.

FIG. 33 depicts an LRU cache, a dirty LRU list, a disk elevator, and process queues forming part of an RNS system.

FIG. 34 depicts the operation of a compressed LRU cache.

FIG. 35 illustrates basic physical components of a disk drive.

FIG. 36 depicts the operation of an extensible hash in a disk elevator.

FIG. 37 illustrates some of supported block sizes for Linux for various filesystem types.

FIG. 38 illustrates some of supported cluster sizes for Windows NT and 2000 for various filesystem types.

FIG. 39 depicts a method of performing read-ahead on a RAID 1 system.

FIG. 40A depicts a read-ahead method using read requests as input.

FIG. 40B depicts a read-ahead method using filesystem knowledge and read requests as input.

FIG. 41 depicts one method of dynamic block stripe allocation.

FIG. 42 depicts an RNS client system that includes the Linux operating system.

FIG. 43 depicts an RNS client system that includes the Windows NT or Windows 2000 operating systems.

FIG. 44 depicts another RNS system that includes the Linux operating system.

FIG. 45 depicts a system combining four RNS systems through an SCI switch to a client.

FIG. 46A depicts a simple NORMA parallel computing system.

FIG. 46B depicts a simple NUMA parallel computing system.

FIG. 46C depicts a simple CCNUMA parallel computing system.

Reference will now be made in detail to some embodiments of the inventions, example of which are illustrated in the accompanying drawings.

DEFINITIONS

For the benefit of the reader, an explanation of some terms used herein is provided.

RAID is an acronym for Redundant Array of Inexpensive (or Independent) Disks. RAID systems combine several drives together for at least one of several purposes. The first purpose is to combine the data space on the drives into a single file system space, creating a virtual disk that is larger than the component disk drives. The second purpose is to provide redundancy of storage, whereby data is stored on more than one disk to provide for recovery in the event of data corruption, disk failure or disk inaccessibility. The third purpose is related to the second, that purpose being to provide additional data for error correction. In that kind of RAID system, additional data is kept in the form of parity or hamming error codes, permitting the recovery of the originally stored data should any one disk drive fail. The fourth purpose is to improve the data throughput of the disk array, essentially by summing the throughput of each component drive.

In the industry there have become defined several levels of RAID systems. The first level, RAID-0, combines two or more drives to create a larger virtual disk. In the dual drive RAID-0 system is illustrated in FIG. 2A one disk 200 contains the low numbered sectors or blocks and the other disk 202 contains the high numbered sectors or blocks, forming one complete storage space. RAID-0 systems generally interleave the sectors of the virtual disk across the component drives, thereby improving the bandwidth of the combined virtual disk. Interleaving the data in that fashion is referred to as striping. RAID-0 systems provide no redundancy of data, so if a drive fails or data becomes corrupted, no recovery is possible short of backups made prior to the failure.

RAID-1 systems include one or more disks that provide redundancy of the virtual disk. One disk is required to contain the data of the virtual disk, as if it were the only disk of the array. One or more additional disks contain the same data as the first disk, providing a `mirror` of the data of the virtual disk. A RAID-1 system will contain at least two disks, the virtual disk being the size of the smallest of the component disks. A disadvantage of RAID-1 systems is that a write operation must be performed for each mirror disk, reducing the bandwidth of the overall array. In the dual drive RAID-1 system of FIG. 2B, disk 204 and disk 206 contain the same sectors or blocks, each disk holding exactly the same data.

RAID-2 systems provide for error correction through hamming codes. The component drives each contain a particular bit of a word, or an error correction bit of that word. FIG. 2C, for example, illustrates a RAID-2 system having a constituent word of 4 bits and 3 hamming code error correction bits. Disks 208, 210, 212, and 214 contain bits 0, 1, 2, and 3 of each word of storage, while disks 216, 218 and 220 contain the hamming error correction bits. RAID-2 systems automatically and transparently detect and correct single-bit defects, or single drive failures, while the array is running. Although RAID-2 systems improve the reliability of the array over other RAID types, they are less popular than some other systems due to the expense of the additional drives, and redundant onboard hardware error correction.

RAID-4 systems are similar to RAID-0 systems, in that data is striped over multiple drives, as exemplified by the three disk RAID-4 system of FIG. 2D. The storage spaces of disks 222 and 224 are added together in interleaved fashion, while disk 226 contains the parity of disks 222 and 224. RAID-4 systems are unique in that they include an additional disk containing parity. For each byte of data at the same position on the striped drives, parity is computed over the bytes of all the drives and stored to the parity disk. The XOR operation is used to compute parity, providing a fast and symmetric operation that can regenerate the data of a single drive, given that the data of the remaining drives remains intact. RAID-3 systems are essentially RAID-4 systems with the data striped at byte boundaries, and for that reason RAID-3 systems are generally slower than RAID-4 systems in most applications. RAID-4 and RAID-3 systems therefore are useful to provide virtual disks with redundancy, and additionally to provide large virtual drives, both with only one additional disk drive for the parity information. They have the disadvantage that the data throughput is limited by the throughput of the drive containing the parity information, which must be accessed for every read and write operation to the array.

RAID-5 systems are similar to RAID-4 systems, with the difference that the parity information is swiped over all the disks with the data, as exemplified by the three disk system of FIG. 2E. Disks 228, 230, and 232 each contain data and parity in interleaved fashion. Distributing the parity data generally increases the throughput of the array as compared to a RAID-4 system. RAID-5 systems may continue to operate though one of the disks has failed. RAID-6 systems are like RAID-5 systems, except that dual parity is kept to provide for normal operation if up to the failure of two drives. An example of a RAID-6 system is shown in FIG. 2F. Disks 234, 236, 238, and 240 each contain data and two parity words labeled P and R.

Combinations of RAID systems are also possible. For example, FIG. 2G illustrates a four disk RAID 1+0 system providing a concatenated file system that is also redundant. Disks 242 and 244 are mirrored, as are 246 and 248. The combination of 242 and 244 are added to 246 and 248 to form a storage space that is twice the size of one individual drive, assuming that all four are of equal size. Many other combinations of RAID systems are possible.

Many implementations of RAID controllers exist, two types being typical. The first common RAID controller, exemplified in FIG. 3A, is hardware based, having a disk controller interfaced to a computer and several other disk controller interfaced to two or more disk drives. A processor, or other device requiring access to a RAID virtual disk is connected to an adapter 302 through an interface. Interface 304 may be a bus interface, such as PCI, or it may be a disk interface, thereby connecting the adapter as a standard disk drive. Common interfaces of today's computers are the IDE interface and the SCSI interface. Adapter 302 interfaces to disks of the RAID array, shown as 310 and 312, through other disk interfaces, shown as 306 and 308. Adapter 302 contains logic or other RAID means to present the array of disks to the processor 300 as a single disk. Hardware-based RAID controllers have the advantage of speed. The second common RAID controller is software based, exemplified in FIG. 3B. In that case, the controller is a part of the operating system of the computer, the computer having interfaces to two or more drives. A processor 314 contains software to perform RAID functions and also has disk interfaces, shown as 316 and 318, to which two or more disks, shown as 320 and 322, are accessible to the RAID software. Software based controllers are more economical than hardware based controllers, but consume a portion of the processor capacity to perform the RAID functions.

The following definitions are common to the field of parallel computing and will be pertinent to some implementations of the invention herein. A NORMA (NO Remote Memory Access) system, illustrated in FIG. 46A, lacks direct hardware facilities for the access of one processor's memory by another processor in an interconnection network. Most computers fall into this category. Parallel computing may be performed over a network 2600 by message passing, provided that the processor 4604 is connected by a network interface 4604 to the network 4600. Those systems maintain all memory locally 4602. NORMA systems may provide access of memory by software applications and drivers. NORMA systems are generally unsuited for parallel computing applications due to the low bandwidths and high latencies of data transfers between processors. NUMA (Non-Uniform Memory Access) systems, illustrated in FIG. 46B, provide facilities for memory access of one processor to another by hardware. NUMA systems, in their fashion, present to software shared memory such that an application may access memory attached to a distant processor as if it were local to the processor on which the application is executing. The characteristics of a NUMA system are generally a processor 4624 having local memory 4622, but also having access to shared memory 4620 by way of a shared memory controller. The shared memory 4620 is virtual, in that it appears to be a distinct entity to processors 4624, but is actually maintained at processors in the parallel network. NUMA systems in general do not provide a memory cache of the shared memory, thus each memory access requires the full data of each read or write be passed across the interconnection fabric. CCNUMA (Cache Coherent NUMA) systems, illustrated in FIG. 46C, do provide a hardware cache of shared memory, thus eliminating the need of passing blocks of memory data across the interconnection network when the cache is coherent with the shared memory on the remote processor (and thus is coherent). The characteristics of a CCNUMA system are a processor 4644 having local memory 4642, also having access to shared memory through a shared memory controller 4646, but with caching facilities having the capability of tracking and maintaining the coherency of the cache with the other controllers on the shared memory network.

The Scalable Coherent Interface (SCI) is an interconnection scheme described by the standard IEEE 1596 (1992). SCI was originally intended for parallel computing applications to provide a high-speed communications channels between processors. SCI links are unidirectional to eliminate the need for bus arbitration and inherent latencies. An SCI interconnected system is generally arranged in a ring topology, as shown in FIG. 1A. In such a system data is passed from one processor element or node to the next until the data reaches its intended recipient. A multi-processor system arranged in a ring topology is susceptible to failure should any single failure occur in any communications link or node. To reduce the risk of failure, a multi-processor system may be implemented with a topology of multiple rings, as shown in FIG. 1B. A multi-processor system may also contain a switch, as shown in FIG. 1C, which may permit a part of the multi-processor system to continue operation should a node or a link fail. An SCI topology may also contain two or more axes, forming an interconnection fabric, as shown in FIG. 1D. Providing more than one topological axis also creates redundancy, while making routing of information somewhat more complex.

The SCI definition generally provides for communication of up to 1 Gigabyte per second, although some implementations provide only a portion of that bandwidth. SCI hardware manufacturers have yet to consolidate SCI hardware to a common set of connectors, cables, and operating speeds, so presently SCI systems are mainly composed of a single vendor's proprietary hardware. The SCI standard provides for addressing of up to 65536 nodes, each node providing up to 48 bits of shared address space or up to 281 terrabytes of shared memory. The SCI standard provides several atomic operations, such as read/modify/write, compare/swap, and fetch/add. SCI systems operate without remote memory caches, and thus fall under the NUMA model.

There are a number of manufacturers of SCI hardware, many offering integrated circuits that may be included in an electronic design. Other manufacturers provide adapters with software for computer systems, thereby eliminating the SCI hardware interfacing design for computer systems. A source for PCI-SCI adapter cards is Dolphin Interconnect Solutions of Westlake Village, Calif.

HSM is an acronym for Hierarchical Storage Management. HSM serves the function of removing old, unused data from a quickly accessible system, such as a hard disk, to a slower accessible system, such as a tape drive. HSM thereby ensures that the data residing on the readily accessible hardware is that data likely to be accessed in the near future, with older data being available through slower access in the unlikely event it is needed. HSM may also serve as a backup system.

DASD is an acronym for Direct Access Storage Device, which is any random-access storage device such as a hard disk. DASD devices are an improvement over sequential storage devices, such as tape drives, that must read the preceding data to the desired data to determine its location on the media.

SISCI is an acronym for Software Infrastructure for SCI, which is an API for the development of applications that use SCI.

DETAILED DESCRIPTION

One embodiment of the invention includes a RAID subsystem having two planes, as exemplified by the system of FIG. 4A. Three or more disks form RAID-5 arrays in the X-plane, one example including 5 disks shown as 401,402, 403, 404, and 405. Each of the disks of the X-plane RAID arrays are located in a unique Y-plane location, an example of a Y-plane being 402, 412, 422,432, and 442. Each X-plane array is included as a virtual disk in the Y-plane, forming a RAID array at levels 0, 1, 1+0, 5, or other configuration as will be understood by those skilled in the art. Y-plane parity disks 441, 442, 443, and 444 provide for redundancy of data of the disks located in the same Y-axis, thereby forming a dual X-Y fault tolerant fabric preferably composed of inexpensive IDE and/OR SCSI disk devices. In a preferred embodiment, disks 405, 414, 423, and 432 contain the X-plane checksum data. FIG. 4B illustrates a related embodiment to that of FIG. 4A, wherein HSM Disk cache storage segment journals are provided as 406, 407, 416, 417, 426, 427, 436, and 437 to tape drives 408, 418, 428, and 438.

A related embodiment of the system of FIG. 4B is shown in FIG. 4C. Drives 450, 451, 452, 453, and 454 are combined in a RAID array, as are drives 460, 461, 462, 463, and 464. Drives 455, 456, 457458, and 459 are configured as mirrors in RAID-1 fashion to drives 450-454, as are drives 465, 466, 467, 468, and 469 to drives 460-464. Parity drives 478, 479, 480, 481, and 482 provide a Y-axis checksum for the entire array, for example 478 contains parity for drives 450, 455, 460, and 465. HSM disk cache storage segment journals 470 and 471 and tape drives 474 and 475 are provided for the cluster of drives 450-454 to provide HSM functions. Likewise HSM cache storage segment journals 472 and 473 and tape drives 476 and 477 are provided for the cluster of drives 460-464. A parity disk, such as 482, may be omitted from the system, if a connection is not available, as the drive adds only minimal amounts of redundancy to the system.

Another related embodiment of the system of FIG. 4B is shown in FIG. 4D. Drives 490a, 490b, 490c, 490d and 490e are combined in a RAID array. Drives 491a, 491b, 491c, 491d and 491e are configured as mirrors in RAID 1 fashion to drives 490a-e, as are drives 492a, 492b, 492c, 492d and 492e, and 493a, 493b, 493c, 493d and 493e to the same drives 490a-e. Parity drives 496a, 496b, 496c, 496d and 496e provide a Y-axis checksum for the entire array, for example 496a contains parity for drives 490a, 491a, 492a and 493a. HSM disk cache storage segment journals 494a and 494b are provided for the cluster of drives 490a-490e to provide HSM functions. A parity disk, such as 496e, may be omitted from the system, if a connection is not available, as the drive add only minimal amounts of redundancy to the system.

In addition to supporting traditional RAID 1+0 and RAID 5 capabilities, the architecture of this embodiment provides dual planes of fault tolerance for RAID arrays. Disk devices are organized into X and Y planes. Both the X and Y planes employ parity check disks that decrease the probability that the failure of one or more disks within a single array will cause permanent data loss or system down time. With this architecture it is possible to lose an entire array of RAID 5 storage and rebuild it dynamically using the Y plane of parity check disks. The example of FIG. 5 illustrates how a dual X-Y plane system can regenerate data blocks in several directions at once in the event a large number of disks fail in the array, or are taken offline. In that figure drives containing parity for the X-axis are shown as P and for the Y-axis as p. Even though a large quantity of drives at various locations have been made unavailable, as shown with an X, the data of all unavailable drives may be recovered through the redundant parity drives. For example, even through drive 520 is unavailable, the data of that drive may be regenerated through a reverse parity calculation from drives 500, 510, 530, and 540.

The dual X-Y plane allows parity XOR generation to occur in two planes at the same time, creating a second parity check disk set. The array is presented to the external host as the same number of disk arrays of RAID 5 storage as X planes of the array. Each RAID 5 array has an additional string of parity disks on both the X and Y planes of the RAID controller. There is a performance penalty of one additional I/O operation to update the Y plane parity disk over a traditional RAID 5 array using this model. The Y-plane in this model may be implemented as a RAID 4 array with fixed positional check disks.

The architecture of that embodiment supports RAID 0+1 X plane configurations of mirrored stripes and striped mirrors. These configurations may also combine the RAID 4 Y plane with RAID 1+0, providing increased fault tolerance and reliability, while providing the performance advantages inherent in RAID 1+0 mirroring implementations over RAID 5. Due to the nature of parity-based RAID, this design allows the size of the RAID 5 arrays to be practicably larger than five disks. On traditional RAID 5 designs, including more than five disks dramatically increases the probability of multiple devices taking down the entire array. RAID 5 and RAID 1+0 configurations might use up to eight disks for mirroring, striping and parity checking in a preferred embodiment.

RAID 1+0 configurations can be combined to create more than simple two-way mirroring. The architecture of one embodiment having four X planes supports four-way mirroring, and up to eight-way distributed mirroring. A configuration might also use three-way RAID 1 mirroring, having the advantage of providing break-away mirror sets that can be split away from an active mirror group, allowing backups to be performed with a snapshot of the data set.

The use of three and four-way mirroring may increase the read performance by performing round-robin reads (load balancing) across mirror sets for incoming requests. This level of mirroring is usually not necessary for fault tolerance, but in cases where large volumes of read only data, such as web pages, are being heavily accessed, it does provide improved read performance when combined with intelligent data striping for RAID 0 data sets.

Use of RAID 4 Y-parity planes allows RAID 1+0 configurations to recover from failures of multiple mirrored devices, increasing the fault tolerance over traditional RAID 1+0 array configurations. The example of FIG. 6 shows failures in an array of RAID 1+0 devices, as was shown in FIG. 4G. Drives 600-604 and 620-624 form two raid arrays, with mirror drives 610-614 duplicating the data of drives 600-604 and mirror drives 630-634 duplicating the data of drives 620-624. Y-axis parity disks 640-644 contain parity for the drives in the same Y-axis, for example drive 640 containing parity for drives 600, 610, 620, and 630. In the example of FIG. 6, failures of both the primary and secondary mirrored devices in a RAID 1+0 array can be recovered from the Y plane RAID 4 parity check disks in real time. The failure of disks 600 and 610, for example, can be recovered because 600 and 610 contain duplicate data and 620/630 and parity disk 640 contain the remaining data for reconstruction.

In another embodiment shown in FIG. 7, a RAID array 700 is connected to a controller 702 through a RAID interface 704, which may be a number of disk controllers or other means of providing communication from controller 702 to the storage of raid array 700. A network interface 706 is provided in controller 702 through which data on RAID array 700 may be accessed by one or more clients shown as 708a, 708b, and 708c. In one embodiment, interface 706 communicates using the SCI standard. In embodiment, interface 706 is an Ethernet interface, and may facilitate array 700 being presented as a network drive over the NFS, NCP SMBFS, iSCSI or other protocols. In another embodiment, both Ethernet and SCI interfaces are provided. For the purposes of this writing, such a system having a RAID array, a controller, and a network interface will be called a RAID network subsystem (RNS).

In one embodiment, an RNS provides on-demand storage journalling capability, hotfix redirection, mirrored caching, annotated storage journalling, dynamic stripe block allocation, dynamically added stripe and mirror sets, break-away mirroring, and infinite HSM storage journalling. In that embodiment the RNS provides network attached storage capabilities via the remote NFS (Network File System), iSCSI (Internet SCSI TCP remote device support) and Novell NCP (NetWare Core Protocol) networking protocols. Those RNS arrays can also attach to an Ethernet network and function as network attached storage. Systems of those RNS arrays can be interconnected over the SCI (Scalable Coherent Interface) clustering interconnect fabric, one such configuration illustrated in FIG. 45. RNS systems 4502, 4504, 4506, and 4508 are connected through an SCI network to provide a RAID storage network. A system 4500 having an SCI interface is connected to the RAID storage network, optionally through an SCI switch 4510, whereby the storage of the RAID storage network may be accessed. Host systems may use an SCI adapter to interface to such an RNS fabric and communicate with individual RAID systems.

In both networked configurations, cabinets formed from those RNS systems can be utilized by multiple host systems, providing a highly scalable storage area networking fabric that supports distributed RAID-1 mirroring and dynamic fail over capability between RNS RAID controllers. Employing the SCI interface at the host system provides high bandwidth, low latency data transfers of block based memory between those RNS systems and their local caches, without the copying overhead typical of LAN based storage.

In that embodiment, an RNS is housed in an expansion cabinet mountable to a 19 inch rack by slide rails that permits up to eight of those RNS devices to be inserted into a chassis forming a single DASD assembly. Drive bays are accessible by sliding the unit out of the cabinet and adding or removing drives into empty drive bays in the unit. The cabinet assembly allows hot swappable drives to be added to the RNS expansion cabinet. In that embodiment, shown in FIG. 8, each RNS expansion cabinet is equipped with two redundant power supplies 802a and 802b having two cooling fans each, two PCI-SCI interface adapters 804a and 804b providing dual ring or single ring SCI topologies, a Pentium II or Pentium III mezzanine card 806 with a PCI bus 808, two SCSI II on-board controllers 810a and 810b, four 3 Ware single-ended 8-way IDE controllers or four SCSI II controllers 812a, 812b , 812c and 812d, either 256 or 512MB of RAM, 3280GB IDE or SCSI disk drives 814 (each individually labeled 815), a 10-100-1000 Ethernet interface 816, and a 19 inch cabinet chassis 800, two serial ports 818a and 818b, either 4MB or 8MB of flash memory 820 to hold the operating system, and four 32GB tape drives. The chassis 800 has two sliding rails, an array of 64 and an array of 8 LED displays for showing disk power or activity mounted to the front panel. HSM segment journal cache disks are optional. Each cabinet supports dual redundant power supplies with two cooling fans each, in the event of a single power supply failure. The mezzanine PII/PIII system board may contain integrated video, serial, and SCSI and IDE support on-board.

The system board also provides standard PCI slots and supports industry standard PCI adapters. Each cabinet provides a PCI-SCI interconnect adapter and a 10-100-1000 Ethernet interface. The operating system control software is booted from flash memory on the system board, and provides a telnet service by default via either the serial port, SCI, or the Ethernet connection over a network for initial configuration of the RNS cabinet.

RNS expansion cabinets can be assembled into strings of SCI based DASD RAID and mirrored storage. In one embodiment, a single DASD cabinet of RNS storage can accommodate eight RNS expansion cabinets, each with 32 80GB disk drives providing 2.56 terabytes, for a total of 20.48 terabytes of available disk storage per vertical DASD cabinet. FIG. 9A shows a front view of such a cabinet 900, and FIG. 9B shows a rear view of that same embodiment. In that embodiment each expansion cabinet 901 has drive indicator LEDs 902 indicating power and disk activity for a particular disk device in the array. Within a DASD cabinet RNS expansion cabinets are interconnected via SCI cables through two SCI ports, 904a and 904b. Optionally, each RNS expansion cabinet can also be connected to a standard Ethernet network through Ethernet port 906 and accessed remotely via TCP/IP, IPX/SPX, or UDP (NFS). In that embodiment, each of the eight RNS expansion cabinets have a parallel port 908, two serial ports 910a and 910b, two cooling fans 912a and 912b, and power receptacles 914a and 914b for dual power supplies.

RNS DASD cabinets can be interconnected via SCI into very large networks of thousands of RNS nodes, as shown in FIG. 10. DASD strings of RNS expansion cabinets 1010, 1012, and 1014 are linked by an SCI network pivoting around SCI switch 1008. SCI devices 1000, 1002, 1004, and 1006 access DASD strings 1010, 1012 and 1014 through the SCI network. Each of DASD strings 1010, 1012 and 1014 is composed of DASD cabinets labeled 1001. An Ethernet connection 1016 is provided by string 1010 in order to provide access to external devices to the DASD string through an Ethernet network. An RNS expansion cabinet can also be configured as a "head of string" and combine the adjacent RNS cabinets into large logical arrays of mirrored storage that are segregated into distinct SCI rings and ringlets. This provides the advantage of allowing host systems the ability to access several strings of SCI storage in tandem. If one of the strings fails, the SCI attached host can fail over to a mirrored string of DASD. DASD string mirroring is another level of mirroring provided by this invention and may be implemented at the SCI host's system adapter software allowing an SCI attached host system to select and mirror to one or more DASD strings in a distributed fashion.

RNS DADS cabinets or DASD strings can also be used as large Network Attached Storage servers to provide Ethernet based storage area networks. An RNS can also be configured to host multiple NFS, NCP, SMBFS, or iSCSI clients via an Ethernet connection, as shown in FIGS. 11 and 12. In FIG. 11, DASD strings 1108, 1110, and 1112 are connected to an Ethernet network. As in FIG. 10, DASD strings 1108, 1110 and 1112 is composed of DASD cabinets labeled 1101. A client 1100 accesses the DASD strings through the NCP protocol. Another client 1102 accesses the DASD string through the NFS protocol. A third client 1104 accesses the DASD strings through the iSCSI protocol. A fourth client 1105 accesses the DASD strings through the NCP, NFS, and iSCSI protocols. In FIG. 12, a DASD string 1208 (again, composed of DASD cabinets labeled 1201) is made available through Ethernet connections 1200, 1202, 1204, and 1206 to a client. The first connection 1200 accesses the string 1208 using the iSCSI protocol, the storage being accessible under/dev/sda and /dev/sdb. The second connection 1202 accesses the string 1208 using the NCP protocol, the storage being accessible under SYS:/ and VOL1:/. The third connection 1204 accesses the string 1208 using the NFS protocol, the storage being accessible under /mnt/remote. The fourth connection 1206 also accesses the string 1208 using the NFS protocol, the storage being accessible under /mnt/remote1.

Host systems utilizing an RNS may attach to the target RNS units via an SCI interconnect fabric. In one embodiment, RNS controllers expose configured units of storage as virtual disks to host systems, and then the hosts system maps these virtual disks into local device handles and presents them to the host systems as local hard disk devices. The actual methods used to create these mappings differ depending upon the host operating system. FIG. 13 pictorially illustrates how groups of RAID 1 and RAID 5 storage can be combined to map mirrored or striped storage into virtual disk windows that are exposed to SCI attached host systems. The host adapter can be configured to be aware of multiple RNS arrays, and perform automatic fail over to known RAID 1 mirrors in the event an entire RNS expansion cabinet fails. Host adapters maintain a mirror map of configured DASD strings, along with head of string location. In this architecture, it is the host adapter that elects the DASD "head of string" by initiating communication to it.

FIG. 13 illustrates a configuration providing a DASD string with failover capability. A RAID 5 array 1302 is first provided as main storage. RAID 5 mirrors 1304 and 1306 mirror 1302 in RAID 1 fashion and are connected in a DASD string. An SCI RAID controller 1300 is configured to select one system such as 1302 as a "head of string" and perform logical RAID 1 mirroring of RAID5 array storage over the SCI fabric. The SCI mount point keeps a table of all identified mirror arrays, and in the event an array box fails, failover dynamically occurs to a mirrored system. Use of this type of configuration allows the creation of cascaded mirror RAID groups, each being capable of electing a new head if an array box fails. For example, 1302 of the string including 1304 and 1306 is configured to be the default head of string. If 1302 becomes unable to communicate with controller 1300, controller 1300 may select a new head of string of either 1304 or 1306. This figure illustrates how a breakaway mirror may be included to permit ease of backups. Provided all of 1302, 1304, and 1306 are functioning in the DASD string, any one of the three can be removed from the string while providing redundant RAID 1 storage. Separate RAID 5 array 1308 may also be controlled by controller 1300 while also controlling a DASD string such as formed by 1302, 1304 and 1306.

This model simplifies array fail over. Head of string is defined as the RNS array the host is configured to send messages to for a particular array of storage. If the head of string fails, the host adapter switches to the next configured array, since they are all accessible on an SCI switched fabric, and resumes normal operations without any noticeable interruption to connected users who are accessing this storage.

By default, Linux maps RAID devices into the /dev/mdXX device layer as virtual disks, and provides a logical volume manager (LVM) interface that allows "pluggable" RAID device mappings. There are also several reserved blocks of RAID device major:minor numbers assigned to DAC, Compaq, and several other vendors of RAID hardware adapters. Any of these device numbers are available to create device nodes for virtual disks in Linux. Linux can support up to 700 disks on a single host system at present, but not all of these device handles will work well with RAID.

FIG. 14 illustrates logically a system having multiple RAID disks to a Linux system with an SCI interface. An RNS 1402 of 12 disks provides three RAID 5 virtual disks, 1414, 1416 and 1418, which are mapped to /dev/md0, /dev/md1 and /dev/md2. Another RNS 1406 of 12 disks provides one RAID 1+0 virtual disk 1420 as /dev/md3. A third RNS 1404 of 12 disks provides three RAID 5 virtual disks, 1432, 1434 and 1436, as /dev/md4, /dev/md5 and /dev/md6. A fourth RNS 1408 of 12 disks provides one RAID 1+0 array 1430 and one RAID 5 array 1428 as /dev/md7 and /dev/md8. Two additional RNS devices 1410 and 1412 are linked in distributed fashion to provide RAID 5+1 virtual disks 1422, 1424, and 1426 as /dev/md9, /dev/md10 and /dev/md11.

Mapping to Linux is relatively simple, and requires that a table of functions be passed to a disk registration function. These functions then handle read and write requests to the device. This is implemented as an SISCI module that loads into the Linux kernel, and periodically receives broadcast packets from attached RNS arrays advertising exported virtual disk devices. The major device number is also passed to this function during registration.

Device objects in Linux register via a call to register_blkdev( ) and pass a table with the attached functions. The SISCI virtual disk driver implements these functions for each registered virtual disk device. The following C language code fragment illustrates one possible set of functions and a data structure for doing so:

    ssize_t mprs_drv_read(struct file *, char *, size_t, loff_t *);
    ssize_t mprs_drv_write(struct file *, const char *, size_t, loff_t *);
    unsigned int mprs_drv_poll(struct file *, poll_table *);
    int mprs_drv_ioctl(struct inode *, struct file *, unsigned int, unsigned
    long);
    int mprs_drv_open(struct inode *, struct file *);
    int mprs_drv_close( struct inode *, struct file *);
    struct file_operations mprs_drv_driver_ops =
    {
        NULL,           /* lseek  */
        mprs_drv_read,  /* read  */
        mprs_drv_write, /* write  */
        NULL,           /* readdir  */
        NULL,           /* poll  */
        mprs_drv_ioctl, /* ioctl */
        NULL,           /* mmap  */
        mprs_drv_open,  /* open*/
        NULL,           /*flush*/
        mprs_drv_close, /* release  */
        NULL,           /* fsync  */
        NULL,           /* fasync  */
        NULL,           /* check_media_change  */
        NULL,           /* revalidate */
        NULL            /* lock  */
        }

    Device type       Major numbers
    IDE (hda-hdt)     0x0300, 0x0340, 0x1600, 0x1640, 0x2100, 0x2140,
                      0x2200, x2240, 0x3800, 0x3840, 0x3900, 0x3940,
                      0x5800, 0x5840, 0x5900, 0x5940, 0x5A00, x5A40,
                      0x5B00, 0x5B40
    SCSI 0-15         0x0800, 0x0810, 0x0820, 0x0830, 0x0840, 0x0850,
    (sda-sdp)         0x0860, 0x0870, 0x0880, 0x0890, 0x08a0, 0x08b0,
                      0x08c0, 0x08d0, 0x08e0, 0x08f0
    SCSI 16-127       0x4100-0x4700
    (sdq-sddx)
    MCA EDSI          0x2400, 0x2440
    (eda, edb)
    XT disks          0x0D00, 0x0D40
    Acorn MFM disks   0x1500, 0x1540
    Parallel Port ATA 0x2F00, 0x2F01, 0x2F02, 0x2F03
    disks
    Software RAID     0x0900, 0x0901, 0x0902, 0x0903, 0x0904, 0x0905,
    groups            0x0906, 0x0907, 0x0908, 0x0909, 0x090a, 0x090b,
                      0x090c, 0x090d, 0x090e, 0x090f
    I2O 0-127         0x5000-0x57f0
    (hda-hddx)
    COMPAQ            0x4800-0x48F0
    SMART2 Array
    0-127 (ida0a-ida7p)
    DAC960 Raid Array 0x3000-0x37F0
    0-15 (ida0a-ida7p)

    /*++
    Routine Description:
     This is the initialization routine for the TRG NetDisk virtual disk device
     driver for W2K.
     This routine creates the device object for the device and performs all
     other driver initialization.
    Arguments:
     DriverObject - Pointer to driver object created by the system.
    Return Value:
     NTSTATUS - The function value is the final status from the
     initialization operation.
    --*/
    {
     NTSTATUS Status;
     UNICODE_STRING SymbolicLinkName;
     UNICODE_STRING DeviceName;
     //
     // Create the device object.
     //
     RtlInitUnicodeString( &DeviceName,
     L".backslash..backslash.Device.backslash..backslash." NET_DISK_DEVICE_NAME
     );
     Status = IoCreateDevice( DriverObject,
            0,
            &DeviceName,
            FILE_DEVICE_NETWORK_DISK,
            0,
            FALSE,
            &NetDiskDeviceObject);
     if (!NT_SUCCESS( Status )) {
      return Status;
     }
     //
     // Initialize the driver object with this driver's entry points.
     //
     DriverObject->MajorFunction[IRP_MJ_CREATE] =
    (PDRIVER_DISPATCH)NetFsdSuccess;
     DriverObject->MajorFunction[IRP_MJ_CLOSE] =
    (PDRIVER_DISPATCH)NetFsdSuccess;
     DriverObject->MajorFunction[IRP_MJ_READ] =
    (PDRIVER_DISPATCH)NetFsdReadWrite;
     DriverObject->MajorFunction[IRP_MJ_WRITE] =
    (PDRIVER_DISPATCH)NetFsdReadWrite;
     DriverObject->MajorFunction[IRP_MJ_CLEANUP] =
    (PDRIVER_DISPATCH)NetFsdSuccess;
     DriverObject->MajorFunction[IRP_MJ_DEVICE_CONTROL] =
    (PDRIVER_DISPATCH)NetFsdDeviceControl;
     DriverObject->MajorFunction[IRP_MJ_SHUTDOWN] =
    (PDRIVER_DISPATCH)NetFsdSuccess;
     //
     // Initialize the global data structures
     //
     ExInitialize FastMutex( &NetDiskMutex );
     NetDiskCount = 0;
     NetDiskDeviceObject->Flags &= .about.DO_DEVICE_INITIALIZING;
     //
     // Create a symbolic link so that the user mode program can get a handle
     // to the just created device object.
     //
     RtlInitUnicodeString( &SymbolicLinkName,
     L".backslash..backslash.??.backslash..backslash." NET_DISK_DEVICE_NAME);
     Status = IoCreateSymbolicLink( &SymbolicLinkName,
     &DeviceName );
     return Status;
    }

    if ((Device = CreateFile( NET_DISK_DEVICE_NAME_W32,
                         GENERIC_READ .vertline. GENERIC_WRITE,
                         0,
                         NULL,
                         OPEN_EXISTING,
                         0,
                         0 )) == INVALID_HANDLE_VALUE)
    {
     printf("Error %d attempting to open the device..backslash.n",
     GetLastError());
     return;
     }
     //
     // Now attempt the actual mount.
     //
     if (!DeviceIoControl( Device,
                  (DWORD)IOCTL_DISK_MOUNT_FILE,
                  &MountData,
                  sizeof(MOUNT_FILE_PARAMETERS),
                  DeviceName,
                  MAX_DEV_NAME_LENGTH,
                  &BytesReturned,
                  NULL ))
     {
     printf("Error %d from DeviceIoControl()..backslash.n", GetLastError());
     }
     //
     // Finally, if the user specified a drive letter, set up the symbolic
     link.
     //
     if (!DefineDosDevice(DDD_RAW_TARGET_PATH, DriveLetter,
     DeviceName))
     {
     printf("Error %d attempting to create the drive letter.backslash.n",
     GetLastError());
     }
     return;
    }

                  typedef struct_SEGMENT_TABLE_ENTRY
                  {
                    BYTE SegmentName[16];
                    ULONG LastSegment;
                    ULONG SegmentTimestampSignature;
                    ULONG SegmentRoot;
                    ULONG SegmentSectors;
                    ULONG VolumeClusters;
                    ULONG SegmentClusterStart;
                    ULONG SegmentFlags;
                    ULONG SegmentStateTable[4];
                  } SEGMENT_TABLE_ENTRY;

    #define MIRROR_NEW_BIT                          0x00000001
    #define MIRROR_PRESENT_BIT                      0x00000002
    #define MIRROR_INSYNCH_BIT                      0x00000004
    #define MIRROR_DELETED_BIT                      0x00000008
    #define MIRROR_REMIRRORING_BIT                  0x00000010
    #define MIRROR_READ_ONLY_BIT                    0x00000020
    #define MIRROR_GROUP_ESTABLISHED_BIT            0x00010000
    #define MIRROR_GROUP_ACTIVE_BIT                 0x00020000
    #define MIRROR_GROUP_OPEN_BIT                   0x00040000
    #define MIRROR_GROUP_MODIFY_CHECK_BIT           0x00080000
    #define MIRROR_GROUP_REMIRRORING_BIT            0x00100000
    #define MIRROR_GROUP_CHECK_BIT                  0x00200000
    #define MIRROR_GROUP_READ_ONLY_BIT              0x00400000

• Inventors
  Merkey, Jeffrey Vernon;
• Assignee
  Canopy Group, Inc. (Lindon, UT)
• Published
  Mar-1-2005
• Current US Classes:
  709/214
  709/216
  709/250
  709/251
  711/148
  714/5
  714/6
  718/105
• Application #
  092707
• International Classes
  G06F 011//14; G06F 011//10; G06F 012//16
• Field of Search
  709/201 709/203 709/214 709/216 709/218 709/225 709/226 709/250 709/251 711/5 711/147 711/148 711/162 714/5 714/6 714/7 714/8 718/105
• Examiner
  Peikari; B. James
• Agent
  McCarthy; Daniel P. Parsons Behle & Latimer
• US Patent References:
  5805788
  5862313
  6185639
  6219727
  6393485
  6601138