howto create GFS on DRBD network disk mirroring

Introduction

This howto will cover how to create a DRBD disk set with GFS concurrent disk access setup.

Why DRBD?

DRBD disk mirroring benefit as a shared SAN disk to archive higher service resilience with low cost hardware component. Alternately, it may be configured for backup purpose with it’s flexibility.

Prerequisite

1. RedHat Cluster Suit and GFS packages

cman-2.0.115-1.el5
gfs2-utils-0.1.62-1.el5
gfs-utils-0.1.20-1.el5
kmod-gfs-0.1.34-2.el5

2. drbd83 package

yum install drbd83

3. Create a partition on both node with using same size, we use [sda5] for this show case.

Disk /dev/sda: 499.5 GB, 499558383616 bytes
255 heads, 63 sectors/track, 60734 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        6540    52428127+  8e  Linux LVM
/dev/sda3            6541        7062     4192965   82  Linux swap / Solaris
/dev/sda4            7063       60734   431120340    5  Extended
/dev/sda5            7063       19221    97667136   83  Linux

Prepare the configuration file /etc/drbd.conf

global {

 # Participate in DRBD’s online usage counter <URL:http://usage.drbd.org>
  usage-count no;
}
common {

# Protocol A: write IO is reported as completed, if it has reached local disk and local TCP send buffer.
# Protocol B: write IO is reported as completed, if it has reached local disk and remote buffer cache.
# Protocol C: write IO is reported as completed, if it has reached both local and remote disk.

   Protocol C;

}

# Define the first disk name "r0"
resource r0 {
startup {

# Wait for connection timeout. The  init  script  drbd blocks the boot process until the DRBD resources are connected.  When the cluster manager starts later, it does not see a resource with internal split-brain.  In case you want to limit the wait time, do it here.  Default is 0, which means unlimited. The unit is seconds.

        wfc-timeout 20;

# Wait for connection timeout, if this node was a degraded cluster.  In case a degraded cluster (= cluster with only one node left) is rebooted,  this  timeout value is used instead of wfc-timeout, because the peer is less likely to show up in time, if it had been dead before. Value 0 means unlimited.

        degr-wfc-timeout     120;

＃ Sets on which  node  the device should be promoted to primary role by the init script. The node-name might either be a host name or the key word both. When this option is not set the devices stay in secondary role on both nodes. Usually one delegates the role assignment to a cluster manager (e.g. heartbeat, GFS).

	#become-primary-on both; ### Remark it during initiate setup, turn this on once the resource disk created.


}

disk {

# In the event of I/O problem, detach it.

	on-io-error detach;

}

net {
# Define the shared secret used in peer authentication.

	cram-hmac-alg "md5";
	shared-secret "$1$Kdk9IEna$1D1.AKIO1nAhF0H86Z1zH0";

# With  this  option  set you may assign primary role to both nodes. You only should use this option if you use a shared storage file system on top of DRBD. At the time of writing the only ones are: OCFS2 and GFS. If you use this option with any other file system, you are going to crash your  nodes  and  to  corrupt your data!

	allow-two-primaries;

# Spit-brain policy for GFS
	after-sb-0pri discard-zero-changes;
	after-sb-1pri discard-secondary;
	after-sb-2pri disconnect;
}

syncer {
	rate 12M; # Transfer rate for drbd, 100Mb ethernet maximum is 12.5M in general.
}

  on node1.localdomain {
    device    /dev/drbd0;
    disk      /dev/sda5;
    address   10.1.10.1:7789;
    meta-disk internal;
  }
  on node2.localdomain {
    device    /dev/drbd0;
    disk      /dev/sda5;
    address   10.1.10.2:7789;
    meta-disk internal;
  }
}

Create DRBD disk resource

On both node1 and node2

drbdadm create-md r0
#Writing meta data...
#initializing activity log
#NOT initialized bitmap
#New drbd meta data block successfully created.

Start DRBD Daemon on both node1 and node2

/etc/rc.d/init.d/drbd start
Starting DRBD resources: [ d(r0) s(r0) n(r0) ]....

Check the role status, you should see both node is Secondary.

drbdadm role r0
Secondary/Secondary

Outdate it on node2 to initiate fresh sync

drbdadm outdate r0

Monitor the syncer status

cat /proc/drbd
version: 8.3.2 (api:88/proto:86-90)
GIT-hash: dd7985327f146f33b86d4bff5ca8c94234ce840e build by mockbuild@v20z-x86-64.home.local, 2009-08-29 14:07:55
 0: cs:SyncTarget ro:Secondary/Secondary ds:Inconsistent/UpToDate C r----
    ns:0 nr:1275904 dw:1275904 dr:0 al:0 bm:77 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:38732684
	[>....................] sync'ed:  3.2% (37824/39068)M

or by command

drbd-overview

Once sync completed, check the role on node1 status again, state changed to Primary

drbdadm role r0
Primary/Secondary

Check Point

With successfully created a /dev/drbd0[r0] and the role changed to Primary on node1, so you’re able to access the drbd drive now and format your partition. Bear in mind when the state changed to “Secondary/Secondary”, that mean you cannot do any I/O operation on that resource.

Turn on Primary role on both node for GFS

Uncomment in /etc/drbd.conf

allow-two-primaries;

or
Enter the following command on both node1 and node2

drbdadm primary r0

Check the role status, you should see both node is Primary now.

drbdadm role r0
Primary/Primary

Format it as GFS

mkfs.gfs -t cluster0:r0 -j 2 /dev/drbd0

Mount it on both node , That’s it!

mount -t gfs /dev/drbd0 /mnt/somewhere