Shurn the Awesomer
High Availability Storage with iSCSI

High Availability Storage with iSCSI

Written on Thu, 3 November 2016

If you have built storage devices, you know that hardware will eventually fail you one day. This is no exception when it comes to cloud storage. Although I recommend using AWS S3, you might have occasions where you just need to use iSCSI for data storage in the cloud and still need the ability to grow storage. Here's a tutorial for that.

You can perform this tutorial on-premise too. You just need to replace the cloud related devices to your physical ones.

I followed many tutorials from many different sites that I could no longer keep track. So here's a thank you to Google for helping me search through the sea of wonderous content to make this tutorial possible. There was very few tutorial on this subject on Ubuntu, let alone Ubuntu 16.04. Most of the tutorials I run across is based on Centos. But fear not, I have overcomed the dragons and I present you this tutorial on Ubuntu 16.04.

Disclaimer: I didn't actually build this on AWS yet. I used virtual machines on my xenserver. Once I have tried it on AWS, I'll update this tutorial and remove this disclaimer.

By the end of this tutorial, you will have:
  • 2 EC2 instances for File Storage with 500GB each, excluding disk for Os
What you need
  • Amazon Web Service account
    • Some knowledge administrating the AWS console
  • Basic linux administration experience
  • Some Networking Knowledge

Preparations


We are going to prepare the materials for the following tutorial.
Setting up Security Group for File Server
First, we create a security group for your file server. Configure accordingly for your use case.

  • Name: Fileserver for Webserver
    • Description: Fileserver
    • Inbound Port: 22
      • Source: Anywhere
    • Inbound Port: 3260
      • Source: Custom - Select the previous Security Group: Webserver you created
    • Inbound Port: 7788
      • Source: Custom - Select the Security Group: Fileserver. This is tricky. You need to create this Group without this rule first, then edit it and add this rule.
    • Inbound Port: 7789
      • Source: Custom - Select the Security Group: Fileserver. This is tricky. You need to create this Group without this rule first, then edit it and add this rule.

Setting up your File Storage


We set up 2 EC2 instances here in 2 different availability zone for high availability.

Step 1: Choose AMI
When launching a new instance, choose Ubuntu 16.04.

Step 2: Choose Instance Type
Instance type: t2.nano (Choose the type that suits your production environment)

Step 3: Configure Instance
Here's where you ensure High Availability. You need to set up your instance such that they exist in different availability zone, so that if 1 zone goes bad, another zone serves as backup. For now, you launch 1 instance in ZoneA. Later, you need to repeat all the steps here for ZoneB.

Step 4: Add Storage
Here's the important part of the tutorial. The AMI you selected uses 8GiB to run. In most cases, this is more than enough to run the OS and the application you want to use. By the end of the tutorial, you will still have about ~6GiB to play around with.

Next, let's add another volume to the instance and give it 500GiB, more if your production environment needs it. This will be the volume for web hosting. For volume type, choose the one that suits your production environment. For tutorial purposes, "Cold HDD (SC1)" is good enough.
Volume Type: Cold HDD (SC1)
Size: 500GiB

Step 5: Tag Instance
Configure the tags you need. I recommend that you name the instance as FS1 for your first instance, and FS2 for your second instance.

Step 6: Configure Security Group
Use the security group you made, Fileserver.

Step 7: Review Instance Launch
Have a final check on the settings and configuration. If everything is good, go ahead and launch the instance. It will take about 5 minutes.

Set up the second file server instance by repeating the steps above. Be sure to launch your instance in ZoneB for HA and name your instance a different name so that you can differentiate easily.

Configuring infinite storage for your file servers

Know your instance IP address name hostname. You will need it later for your configuration. Look into your EC2 details in the AWS console and find the private IP and private DNS. It should look like:

  • FS1
    • IP Address: 10.0.0.2
    • ip-10-0-0-2.ec2.internal
  • FS2
    • IP Address: 10.0.0.3
    • ip-10-0-0-3.ec2.internal


Your ip address will be different from mine. Another important thing to note, your VPC must be able to resolve DNS hostname. Enable it in your AWS console if it isn't. Otherwise, the hostname cannot be resolved in the file servers we are about to configure.

Let's make sure your file server has the latest patches

apt-get update
apt-get -y dist-upgrade
apt-get -y autoremove

After an update, it's usually a good idea to give your instance a reboot

reboot

Look for your attached hard disk of 500GiB.

parted -l



It should look something like these if you've followed the previous steps accordingly.

So our hard disk is /dev/xvdb. We are going to let LVM manage this disk.

pvcreate /dev/xvdb

Next, we are going to create a volume group called "inifinitestoreVG" with the hard disk.

vgcreate infinitestoreVG /dev/xvdb

We can't use all 500GiB because LVM just seems to need some space for itself. 0.1GiB ain't significant when you can infinitely add storage anyway. So for the LUN, we will use 499.9GiB.

lvcreate -n infinitestoreLV -L 499.9g infinitestoreVG


Let's look at your Logical Volumes.

lvdisplay


Configuring DRBD


This section of the tutorial involves both nodes. All commands are executed on both nodes.

We need to install drbd8-utils and pacemaker. We will also use ntp for accurate time sync

apt-get install -y drbd8-utils pacemaker corosync ntp tgt

Avoid the DRBD startup script from being automatically launch at startup

update-rc.d -f drbd remove

We need to configure the hosts.

nano /etc/hosts


Ensure that the entry looks like this:

10.0.0.2 ip-10-0-0-2.ec2.internal ip-10-0-0-2
10.0.0.3 ip-10-0-0-3.ec2.internal ip-10-0-0-3


Your IP and DNS will likely be different, so use yours.

Edit drbd configuration and define the resource.

nano /etc/drbd.d/iscsi.res


Add the following to the file.

resource iscsi {
on ip-10-0-0-2 {
device /dev/drbd0;
disk /dev/infinitestoreVG/infinitestoreLV;
address 10.0.0.2:7788;
meta-disk internal;
}

on ip-10-0-0-3 {
device /dev/drbd0;
disk /dev/infinitestoreVG/infinitestoreLV;
address 10.0.0.3:7788;
meta-disk internal;
}
}


Be sure to replace the hostname and the ip address to match yours.

Initialize the meta-data disk on both servers.

drbdadm create-md iscsi
drbdadm up iscsi


if these commands check with you on whether to destroy existing data or not. Just reply yes. It's a new disk anyway.

Now restart drbd service

service drbd restart

Preparing DRBD for iSCSI target


In this section of the tutorial, all commands are specific to each node and will be indicated.

We will use node1 as the primary for the drbd device that will contain the iSCSI configuration files and initiate first full sync.

[node1] drbdadm -- --overwrite-data-of-peer primary iscsi

Check the progress of sync with:

[node1] cat /proc/drbd

Pacemaker and Corosync Configuration

Generate a key for the cluster to communicate with each other. Be prepared for tired fingers.

[node1] corosync-keygen


Then we copy [node1]/etc/corosync/authkey to the other node either through SFTP or other means to [node2]/etc/corosync/authkey. Ensure it has the right permissions on both nodes.

chmod 644 /etc/corosync/authkey

Now we are going to tell Corosync the cluster we are making on both nodes. In AWS, broadcast is disabled, so we must specify the individual members of the nodes manually. We will also have to use UDPU as the transport protocol.

We open up /etc/corosync/corosync.conf on both nodes.

nano /etc/corosync/corosync.conf


If there is anything existing, leave it as it's default. Your bindnetaddr is the network address, not the node address. Your totem configuration should look something like this:

totem {
version: 2
secauth: off
interface {
member {
memberaddr: 10.0.0.2
}
member {
memberaddr: 10.0.0.3
}
ringnumber: 0
bindnetaddr: 10.0.0.0
mcastport: 5405
ttl: 1
}
transport: udpu
}


Now append the following to the file.

service {
name: pacemaker
ver: 1
}


Save this file and proceed.

Open /etc/default/corosync on both nodes.

nano /etc/default/corosync


Now set this configuration. If it does not exist, just append it to the file.

START=yes

If your drbd sync is complete, it is a good time to reboot the computer for all the settings to take effect. Otherwise, just wait for a bit till the sync finishes.

cat /proc/drbd
reboot

Now let's see Cluster Resource Mamangement (CRM) in action.

crm_mon


You should see that a connection has been established. If it's attempting to connect, somewhere has gone wrong. Restart the tutorial.

We configure the CRM on node 1.

[node1] crm configure


These are the configurations you need to type

[node1] primitive p_drbd_iscsi ocf:linbit:drbd params drbd_resource="iscsi"
[node1] primitive p_target_iscsi ocf:heartbeat:iSCSITarget params implementation="tgt" iqn="iqn.2016-10.me.shurn:disk.0" tid="1" additional_parameters="DefaultTime2Retain=60 DefaultTime2Wait=5" op monitor interval="10"
[node1] primitive p_lu_iscsi_lun1 ocf:heartbeat:iSCSILogicalUnit params lun="1" path="/dev/drbd0" target_iqn="iqn.2016-10.me.shurn:disk.0" op monitor interval="10"
[node1] primitive p_ip ocf:heartbeat:IPaddr2 params ip="10.0.0.4" cidr_netmask="16" nic="eth0"
[node1] ms ms_drbd_iscsi p_drbd_iscsi meta notify="true" master-max="1" master-node-max="1" clone-max="2" clone-node-max="1"
[node1] order o_drbd_before_target inf: ms_drbd_iscsi:promote p_target_iscsi:start
[node1] order o_target_before_lun inf: p_target_iscsi:start p_lu_iscsi_lun1:start
[node1] order o_drbd_before_ip inf: ms_drbd_iscsi:promote p_ip:start
[node1] colocation c_drbd_with_target inf: p_target_iscsi ms_drbd_iscsi:Master
[node1] colocation c_drbd_with_lun inf: p_lu_iscsi_lun1 ms_drbd_iscsi:Master
[node1] colocation c_drbd_with_ip inf: p_ip ms_drbd_iscsi:Master
[node1] property stonith-enabled=false
[node1] property no-quorum-policy=ignore
[node1] commit
[node1] exit


Verify that the changes you have done is in effect

crm status

You are done

If you are building an application that wants to use HA iSCSI instead of standalone iSCSI, such as Xenserver and VDI, you now have a peace of mind.

Navigate
Installing Zimbra with Docker
Multiverse: Science or Fiction?
Related Content