Shurn the Awesomer

Setting up a web hosting site with infinite storage

Written on Thu, 5 May 2016

Running a web hosting site is already quite a challenge for most beginners. What if you're task with running a site that could potentially be hosting a ton of image and videos? Well yeah, you could use S3 storage from Amazon Web Services. But that application just doesn't support S3 storage and modifying the application ain't an option.

So the big question, how do you have a server storage that can continually grow without downtime involved?
Welcome to the world of LVM, Logical Volume Management.

In this tutorial, I'll be showing you how you can setup a Ubuntu 16.04 on AWS with infinite growing storage space, by regularly adding EBS volumes to your instance whenever you need.

By the end of this tutorial, you will have:
  • Ubuntu 16.04
  • 3 EBS volumes attached with 1TiB space available for web hosting
    • 8GiB for OS and softwares
    • 2x 500GiB volume for web hosting, formatted with BTRFS
  • 1 Volume Group
  • 1 Logical Volume
  • LAMP Stack Installed
What you need:
  • Amazon Web Service account
    • Some knowledge administrating the AWS console
  • Basic linux administration experience
What will not be covered:
  • Database tables for infinite growth. Use RDS for that. But if you absolutely need it, the steps are quite similar to setting up apache logs section.

Setting up the instance


Step 1: Choose AMI
Locate the right AMI for your region with the AMI Locator. I'm using ami-840910ee because I want my instance to be located in North Virginia. Try to use 16.04 LTS since this is what the tutorial is all about. For older versions, additional steps are required as you progress through this tutorial, which will not be covered.

Step 2: Choose Instance Type
Instance type: t2.nano (Choose the type that suits your production environment)

Step 3: Configure Instance
Configure accordingly. If you're not sure, usually the default settings will do fine. Read more at AWS documentation.

Step 4: Add Storage
Here's the important part of the tutorial. The AMI you selected uses 8GiB to run. In most cases, this is more than enough to run the OS and the application you want to use. In our case, LAMP stack won't even take up 1GiB. By the end of the tutorial, you will still have about ~6GiB to play around with.

Next, let's add another volume to the instance and give it 500GiB, more if your production environment needs it. This will be the volume for web hosting. For volume type, choose the one that suits your production environment. For tutorial purposes, "Cold HDD (SC1)" is good enough.
Volume Type: Cold HDD (SC1)
Size: 500GiB

Step 5: Tag Instance
Configure the tags you need. It's ok to leave blank if you're not sure.

Step 6: Configure Security Group
Configure the security group. You will definitely need at port 22, 80, and 443. Adjust and add settings as your application needs.

Step 7: Review Instance Launch
Have a final check on the settings and configuration. If everything is good, go ahead and launch the instance. It will take about 5 minutes. So meantime, grab a cup of tea.

Configuring the web host

SSH into your newly launched instance. Your username is "ubuntu". You will need your keyfile that was generated for you to login.

It's a good time to run all the necessary procedures to keep your Ubuntu up to date with the latest security patches. All codes are running with root access for convenience. If you prefer not to run on root, be sure to prepend "sudo" in front of most command.

sudo su
apt-get update
apt-get -y dist-upgrade
apt-get -y autoremove


After an update, it's usually a good idea to give your instance a reboot

reboot now

SSH back into your instance. Let's install the LAMP stack.

sudo su
apt-get -y install lamp-server^


You will be promted to give your mysql root user a password. Give it a password even if you don't intend to use it, as a good practice.

By this point, you can browse to the instance IP address with your web browser. You will see a Apache2 Ubuntu Default Page, with the world's most famous cliche: It works!

If you don't see this, you might want to restart this tutorial or trace back where you might have gone wrong.

Configuring your infinite storage

Finally, the main point of this tutorial. You will need to start off with looking at your attached hard disk.

parted -l



It should look something like these if you've followed the previous steps accordingly.

So our hard disk is /dev/xvdb. We are going to let LVM manage this disk.

pvcreate /dev/xvdb

Next, we are going to create a volume group called "inifinitestoreVG" with the hard disk.

vgcreate infinitestoreVG /dev/xvdb

Finally, we will create a logical volume out of the volume group called "infinitestoreLV" with 499.9GiB. We can't use all 500GiB because LVM just seems to need some space for itself. 0.1GiB ain't significant when you can infinitely add storage anyway.

lvcreate -n infinitestoreLV -L 499.9g infinitestoreVG


Let's look at your Logical Volumes.

lvdisplay


Your new logical volume is /dev/infinitestoreVG/infinitestoreLV.

We will need to format this new volume with a filesystem. We are going to use BTRFS because of its cool features. A word of caution: if you are using older versions of linux, BTRFS might still be experimental. Use ext4 if you are using older ubuntu versions.

mkfs.btrfs /dev/infinitestoreVG/infinitestoreLV

Let's mount the new volume in at location where apache host its websites.

mount /dev/infinitestoreVG/infinitestoreLV /var/www

Right now, it's an empty hard disk. So let's create back the site to show that we can use the new hard disk for hosting.

mkdir /var/www/html
nano /var/www/html/index.php


We are going to make a PHP page so that we can output something.

<?php phpinfo();


By this point, if you have done everything right, you can see a PHP info output on your browser when you browse to your instance's ip address.

Congratulations, you have successfully run a web hosting site on an ever growing volume group whenever you want it to.

But that's not the end yet. What happens when you reboot? You will have to mount the volume group again manually. To automate this process, let's modify the fstab:

nano /etc/fstab


Add the following line:

/dev/infinitestoreVG/infinitestoreLV /var/www/ btrfs defaults 0 0


Now if you reboot and browse back to the IP address, you will still see the php output. If you see the default ubuntu page instead, the auto-mount is not working.

reboot now

Optional: Apache Logs


Generally speaking, you would do fine if you stop right here. But if you want to take one step further, you can also ensure the all apache logs are also located in the Logical Volume. This is very useful if you intend to keep your logs for eternity!
First, create a folder in the Logical Volume:

mkdir /var/www/log


Now, you need to modify the settings of apache to generate logs in the new folder

nano /etc/apache2/envvars


Look for the following line:

export APACHE_LOG_DIR=/var/log/apache2$SUFFIX


Change it to:

export APACHE_LOG_DIR=/var/www/log$SUFFIX


Now, restart apache:

service apache2 restart

Expanding your storage


Has it come to the time to increase your space for web hosting? You have nothing to fear, if you've followed the previous steps correctly.
Let's start with giving yourself a new 500GiB, or more, volume in your AWS console. It is recommended that you keep the volume type consistent will the ones you currently have. I was using "Cold HDD (SC1)" before, so I will use back the same one. You will need to create it in the same zone as your instance.

Next, we need to attach it to the instance.
Device: /dev/sdc

Now go back to the SSH. Let's have a look at the hard disk detected by the OS.

parted -l


You should see "/dev/xvdc".

Let's ensure the device is known to LVM.

pvcreate /dev/xvdc


Let's give the device to infinitestoreVG.

vgextend infinitestoreVG /dev/xvdc


Extend the volume all the way to 1TiB.

lvextend /dev/infinitestoreVG/infinitestoreLV /dev/xvdc


Last step before we can finally use the new space we added. The file system must recognise the newly added space.

btrfs filesystem resize max /var/www
df -h


Now, you should have 1000G worth of space in /var/www, or 1TiB.

Congratulations


You now have a web hosting server where you can grow your hosting space infinitely. You can attached unlimited amount of EBS volumes to an instance. Just take note that there is a limit of EBS volumes you are allowed to create. And also, BTRFS has a limit of 16EiB, which is 16 384PiB, which is 16 777 216 TiB. You have a limit of 9 223 372 036 854 775 807 files allowed. You're probably not going to reach that limit anyway, so that's not something you need to worry so soon.

You can read up more information about:

  • Logical Volume Management
  • BTRFS: You should really take advantage of the features available in BTRFS for your hosting. Particularly, the quota management feature available.
Navigate
The best teachers
Improving my little site
Related Content