Clustered Filesystems

Under Construction
This document / space is still under construction

This article contains links, resources and information on clustered and distributed filesystems

Redundancy

Does not have
- Redhat GFS, OCFS2, PVFS
Will have
- Lustre
Has
- GPFS, iBrix Fusion, GlusterFS
RedHat GFS provides redundancy for the central lock manager, but has no redundancy built in for the data. File system corruption therefore would require recovery
Lustre also relies on underlying RAID to provide redundancy. If there is filesystem corruption, Lustre relies on ext3 or its own tools to recover. While Lustre works on striping a file across OSSs, this is a simple RAID 0 stripe without any redundancy. It seems however this will be present in Lustre 2.0 which should be available in 2nd quarter 2009

Parameters to judge a filesystem

Does it works on top of an existing file system or is a filesystem itself
How long has it been around
- Lustre 1.0 released in 2003
What layer does it work on?
RedHat GFS operates at the block device layer. It essentially provides a way to mount multiple storage devices across multiple machines
Special Client Library
Symmetric vs Assymmetric
NFS Export
CIFS Export
Quota Support
Redundancy
License
- GPFS - Proprietary and commercial
- ibrix Fusion - Proprietary and commercial
Software only or with Hardware

List of Filesystems

RedHat GFS - http://www.redhat.com/gfs/
Oracle OCFS2 - http://oss.oracle.com/projects/ocfs2/
IBM GPFS - http://www-03.ibm.com/systems/clusters/software/gpfs/index.html
Sun's Lustre - http://www.sun.com/software/products/lustre/index.xml
Gluster - http://gluster.org/
CODA - http://www.coda.cs.cmu.edu/about.html
PVFS - http://www.pvfs.org/
VMWare VMFS - http://www.vmware.com/products/vi/esx/vmfs.html
AFS - http://www.openafs.org/
http://www.panasas.com/activestor.html
DRBD
Ibrix Fusion - http://www.ibrix.com/products-overview.htm
http://www.isilon.com/
http://www.exanet.com/
http://www.rackable.com
Panasas Activescale - http://www.panasas.com

Discussion

GPFS

From: http://www-03.ibm.com/systems/clusters/software/whitepapers/gpfs_intro.html

The IBM General Parallel File System (GPFS) is more than just a file system; it is a file management infrastructure. It provides unmatched performance and reliability with scalable access to critical file data. GPFS distinguishes itself from other cluster file systems by providing concurrent high-speed file access to applications executing on multiple nodes of an AIX cluster, a Linux cluster, or a heterogeneous cluster of AIX and Linux nodes. New with GPFS 3.2.1.5 you can add windows client nodes to an AIX, Linux or mixed cluster. In addition to providing file storage capabilities, GPFS provides storage management, information life cycle management tools, centralized administration and allows for shared access to file systems from remote GPFS clusters.

GPFS provides scalable high-performance data access from a single node cluster, a two node cluster providing a high availability platform supporting a database application, for example, to 2,000 nodes or more used for applications like modeling weather patterns. Up to 512 Linux nodes or 128 AIX nodes with access to one or more file systems are supported as a general statement and larger configurations exist by special arrangements with IBM. The largest existing configurations exceed 2,000 nodes. GPFS has been available on AIX since 1998 and Linux since 2001. It has been field proven time and again on some of the world's most powerful supercomputers1 to provide efficient use of disk bandwidth.

GlusterFS

http://www.gluster.org/docs/index.php/Who%27s_using_GlusterFS

Currently testing glusterfs as alternative to SAN storage. Utilizing a cluster of recycled hosting customer servers, Viawest hopes to save over $500,000 USD compared to commercial vendors like Isilon or Ibrix. Our current deployment features GlusterFS 1.3.12 running on ArchLinux. Projections show that with a small purchase of aprx $15,000 USD on replacement SATA Drives (spanning 30 physical 1U servers), the total possible volume 135TB RAW, and aprx 65TB Useable when implementing AFR. Research and Testing has shown that GlusterFS is more flexible than Lustre or AFS.

DRBD

Whatever is not flushed from the buffer cache will not get copied over to the other disk
DRBD is, by definition and as mandated by the Linux kernel architecture, agnostic of the layers above it. Thus, it is impossible for DRBD to miraculously add features to upper layers that these do not possess. For example, DRBD cannot auto-detect file system corruption or add active-active clustering capability to file systems like ext3 or XFS
For good or bad, drbd mirrors EVERYTHING, so if the filesystem is screwed up on one, then it probably is on the other as well, unless the problem was a physical problem with the disks, or a corruption problem in the device driver (http://www.gossamer-threads.com/lists/drbd/users/12494?search_string=filesystem%20corruption;#12494