Finally! Categories

Written on Thu, 19 May 2016

Ok, so I have finally added categories into my blog so that I can properly categorised my blog post. More than just administrative, it's so that I can put up the menu up there and have you sort out the post. Plus, I like things organised. And if I'm going to get any advertising agencies for my site to earn a tiny income from here, I need to have proper site maps.

There's still a lot of things I want to work on for my little site. The most painful eye-sore is the site design. I really should start working on it. But then again, it's not my highest priority right now.

I'm considering a tick-tock improvement cycle for my site, somewhat similar to how Intel implements their tick-tock model, shifting from working on shrinking their transistors to working on a better microarchitecture, and back again. So I'm going to implement something like that, shifting from implementing site improvements on existing items, to implementing something new on my site, which I will share how I did that new thing to the world.

Interestingly, Intel is not going to use that tick-tock model anymore, instead, opting for "process-architecture-optimization" cycle. Moore's law is in trouble.

Right, so on to what I'm going to work on next. Maybe https for my site?

Piwik: Private Analytics

Written on Fri, 13 May 2016

Piwik is easily the best analytics for organisations who have a privacy policy that doesn't allow them to share information with third party. If you've tried Google Analytics, who will find that Piwik offers very similar functionalities. The biggest downside is probably that Google actually integrates its services seamlessly, while piwik is a purely standalone product. But the biggest advantage is that Piwik is entirely open sourced, so you can go ahead to modify or extend its functionalities.

Requirements

Minimum:

• Webserver such as Apache, Nginx, IIS, etc
• PHP5.3 or greater
• MySQL version 4.1 or greater, or MariaDB
• PHP extension pdo and pdo_mysql, or the mysqli extension

Recommended:

• Webserver such as Apache, Nginx, IIS, etc
• PHP7
• Latest MySQL or MariaDB
• PHP extension pdo and pdo_mysql, or the mysqli extension

Tip: Check out my other blog post for more information on setting up a web hosting with PHP7 and infinitely growing hard disk space.

Preparing the Server

Now I assume you have followed my previous blog post and installed PHP7 on Ubuntu 16.04. If not, do check it out! Let's prepare the server for Piwik installation.

Starting with the required libraries:

apt-get install php7.0-curl php7.0-gd php7.0-cli php7.0-dev libgeoip-dev

Interestingly, at the time of writing, GeoIP extension is not supported in PHP7. So we are going to need some help. Fortunately, Zakay from GitHub has compiled a solution. Let's hope there will be an official release of GeoIP extension for PHP7 soon.

We are going to get the codes from github:

git clone https://github.com/Zakay/geoip.git
cd geoip

Next, we compile it:

phpize
./configure --with-php-config=/usr/bin/php-config
make
make install

Now it is ready to be used. Let's put the settings in php.ini so that it will during runtime.

touch /etc/php/7.0/mods-available/geoip.ini
echo "extension = geoip.so"> /etc/php/7.0/mods-available/geoip.ini
phpenmod geoip
service apache2 restart

I hope you still have your phpinfo.php file in your webroot. Run the script in your browser and you should see GeoIP installed.

Next we configure your mysql so that it has a username for itself. It's always a good security measure that you assign each app to a single user and not shared.

mysql -p

Then enter your root password. If you have another username, do use the following command instead:

mysql -u anotherusername -p

You should now be at the mysql console. Enter the following:

CREATE DATABASE piwik_db_name_here;
CREATE USER 'piwik'@'localhost' IDENTIFIED BY 'password';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES ON piwik_db_name_here.* TO 'piwik'@'localhost';
exit

Do substitute the names accordingly. Otherwise, your future administrators might think you're lazy.

Once you are done with that, you are absolutely good to proceed to piwik installation.

Installing Piwik

Now you are ready to download the piwik installation. You can download it to your computer, unzip, and upload it via ftp or sftp. But for pure linux sake, here's the commands to do it:

apt-get install unzip
wget http://builds.piwik.org/piwik.zip
unzip piwik.zip
mv piwik/* /var/www/html
rmdir piwik
rm How\ to\ install\ Piwik.html piwik.zip

The last few commands to remove is for cleaning up no longer needed files. It's always a good habit to clean up after yourself. Keep servers neat and tidy is important. The piwik is installed in the default webroot. If you are using it as a virtual host, you must configure apache accordingly, which is not covered in this tutorial.

If you browse to your website in your web browser, you should see your piwik installation. If the page shows an error /tmp/cache/tracker, you need to ensure that the tmp folder has enough write permission. It's also recommended that you keep the entire installation writable by apache so that piwik can do self-upgrade.

Go ahead to install piwik by following the instructions. And you're done! Congratulations!

Improving my little site

Written on Thu, 12 May 2016

Ok, so I've been spending the past few days trying to improve the site at the back. This is very important as the content that is generated on the front depends a lot on what was at the back. The system behind the blog posts were not correctly rendering <br /> properly. So I had to dig deeper to find out why my /n is not properly converted.

Turns out, the way the wiki text parser is treats every line break as a new line to parse, thus totally ignoring my regex that needs to read the /n symbol. Thus I had to do hackish methods make line breaks work.

It works out now properly.
Like!
This!
Tada!

And the previous posts are now rendering line breaks properly too.

Setting up a web hosting site with infinite storage

Written on Thu, 5 May 2016

Running a web hosting site is already quite a challenge for most beginners. What if you're task with running a site that could potentially be hosting a ton of image and videos? Well yeah, you could use S3 storage from Amazon Web Services. But that application just doesn't support S3 storage and modifying the application ain't an option.

So the big question, how do you have a server storage that can continually grow without downtime involved?
Welcome to the world of LVM, Logical Volume Management.

In this tutorial, I'll be showing you how you can setup a Ubuntu 16.04 on AWS with infinite growing storage space, by regularly adding EBS volumes to your instance whenever you need.

---

By the end of this tutorial, you will have:

• Ubuntu 16.04
• 3 EBS volumes attached with 1TiB space available for web hosting
  • 8GiB for OS and softwares
  • 2x 500GiB volume for web hosting, formatted with BTRFS
• 1 Volume Group
• 1 Logical Volume
• LAMP Stack Installed

What you need:

• Amazon Web Service account
  • Some knowledge administrating the AWS console
• Basic linux administration experience

What will not be covered:

• Database tables for infinite growth. Use RDS for that. But if you absolutely need it, the steps are quite similar to setting up apache logs section.

---

Setting up the instance

Step 1: Choose AMI
Locate the right AMI for your region with the AMI Locator. I'm using ami-840910ee because I want my instance to be located in North Virginia. Try to use 16.04 LTS since this is what the tutorial is all about. For older versions, additional steps are required as you progress through this tutorial, which will not be covered.

Step 2: Choose Instance Type
Instance type: t2.nano (Choose the type that suits your production environment)

Step 3: Configure Instance
Configure accordingly. If you're not sure, usually the default settings will do fine. Read more at AWS documentation.

Step 4: Add Storage
Here's the important part of the tutorial. The AMI you selected uses 8GiB to run. In most cases, this is more than enough to run the OS and the application you want to use. In our case, LAMP stack won't even take up 1GiB. By the end of the tutorial, you will still have about ~6GiB to play around with.

Next, let's add another volume to the instance and give it 500GiB, more if your production environment needs it. This will be the volume for web hosting. For volume type, choose the one that suits your production environment. For tutorial purposes, "Cold HDD (SC1)" is good enough.
Volume Type: Cold HDD (SC1)
Size: 500GiB

Step 5: Tag Instance
Configure the tags you need. It's ok to leave blank if you're not sure.

Step 6: Configure Security Group
Configure the security group. You will definitely need at port 22, 80, and 443. Adjust and add settings as your application needs.

Step 7: Review Instance Launch
Have a final check on the settings and configuration. If everything is good, go ahead and launch the instance. It will take about 5 minutes. So meantime, grab a cup of tea.

---

Configuring the web host

SSH into your newly launched instance. Your username is "ubuntu". You will need your keyfile that was generated for you to login.

It's a good time to run all the necessary procedures to keep your Ubuntu up to date with the latest security patches. All codes are running with root access for convenience. If you prefer not to run on root, be sure to prepend "sudo" in front of most command.

sudo su
apt-get update
apt-get -y dist-upgrade
apt-get -y autoremove

After an update, it's usually a good idea to give your instance a reboot

reboot now

SSH back into your instance. Let's install the LAMP stack.

sudo su
apt-get -y install lamp-server^

You will be promted to give your mysql root user a password. Give it a password even if you don't intend to use it, as a good practice.

By this point, you can browse to the instance IP address with your web browser. You will see a Apache2 Ubuntu Default Page, with the world's most famous cliche: It works!

If you don't see this, you might want to restart this tutorial or trace back where you might have gone wrong.

---

Configuring your infinite storage

Finally, the main point of this tutorial. You will need to start off with looking at your attached hard disk.

parted -l

It should look something like these if you've followed the previous steps accordingly.

So our hard disk is /dev/xvdb. We are going to let LVM manage this disk.

pvcreate /dev/xvdb

Next, we are going to create a volume group called "inifinitestoreVG" with the hard disk.

vgcreate infinitestoreVG /dev/xvdb

Finally, we will create a logical volume out of the volume group called "infinitestoreLV" with 499.9GiB. We can't use all 500GiB because LVM just seems to need some space for itself. 0.1GiB ain't significant when you can infinitely add storage anyway.

lvcreate -n infinitestoreLV -L 499.9g infinitestoreVG

Let's look at your Logical Volumes.

lvdisplay

Your new logical volume is /dev/infinitestoreVG/infinitestoreLV.

We will need to format this new volume with a filesystem. We are going to use BTRFS because of its cool features. A word of caution: if you are using older versions of linux, BTRFS might still be experimental. Use ext4 if you are using older ubuntu versions.

mkfs.btrfs /dev/infinitestoreVG/infinitestoreLV

Let's mount the new volume in at location where apache host its websites.

mount /dev/infinitestoreVG/infinitestoreLV /var/www

Right now, it's an empty hard disk. So let's create back the site to show that we can use the new hard disk for hosting.

mkdir /var/www/html
nano /var/www/html/index.php

We are going to make a PHP page so that we can output something.

<?php phpinfo();

By this point, if you have done everything right, you can see a PHP info output on your browser when you browse to your instance's ip address.

Congratulations, you have successfully run a web hosting site on an ever growing volume group whenever you want it to.

But that's not the end yet. What happens when you reboot? You will have to mount the volume group again manually. To automate this process, let's modify the fstab:

nano /etc/fstab

Add the following line:

/dev/infinitestoreVG/infinitestoreLV /var/www/ btrfs defaults 0 0

Now if you reboot and browse back to the IP address, you will still see the php output. If you see the default ubuntu page instead, the auto-mount is not working.

reboot now

---

Optional: Apache Logs

Generally speaking, you would do fine if you stop right here. But if you want to take one step further, you can also ensure the all apache logs are also located in the Logical Volume. This is very useful if you intend to keep your logs for eternity!
First, create a folder in the Logical Volume:

mkdir /var/www/log

Now, you need to modify the settings of apache to generate logs in the new folder

nano /etc/apache2/envvars

Look for the following line:

export APACHE_LOG_DIR=/var/log/apache2$SUFFIX

Change it to:

export APACHE_LOG_DIR=/var/www/log$SUFFIX

Now, restart apache:

service apache2 restart

---

Expanding your storage

Has it come to the time to increase your space for web hosting? You have nothing to fear, if you've followed the previous steps correctly.
Let's start with giving yourself a new 500GiB, or more, volume in your AWS console. It is recommended that you keep the volume type consistent will the ones you currently have. I was using "Cold HDD (SC1)" before, so I will use back the same one. You will need to create it in the same zone as your instance.

Next, we need to attach it to the instance.
Device: /dev/sdc

Now go back to the SSH. Let's have a look at the hard disk detected by the OS.

parted -l

You should see "/dev/xvdc".

Let's ensure the device is known to LVM.

pvcreate /dev/xvdc

Let's give the device to infinitestoreVG.

vgextend infinitestoreVG /dev/xvdc

Extend the volume all the way to 1TiB.

lvextend /dev/infinitestoreVG/infinitestoreLV /dev/xvdc

Last step before we can finally use the new space we added. The file system must recognise the newly added space.

btrfs filesystem resize max /var/www
df -h

Now, you should have 1000G worth of space in /var/www, or 1TiB.

---

Congratulations

You now have a web hosting server where you can grow your hosting space infinitely. You can attached unlimited amount of EBS volumes to an instance. Just take note that there is a limit of EBS volumes you are allowed to create. And also, BTRFS has a limit of 16EiB, which is 16 384PiB, which is 16 777 216 TiB. You have a limit of 9 223 372 036 854 775 807 files allowed. You're probably not going to reach that limit anyway, so that's not something you need to worry so soon.

You can read up more information about:

• Logical Volume Management
• BTRFS: You should really take advantage of the features available in BTRFS for your hosting. Particularly, the quota management feature available.

The best teachers

Written on Wed, 4 May 2016

Today I came across a very nice quote:

The best teachers are those who show you where to look, but don't tell you what to see.

- Alexandra K. Trenfor

Now, how true that. When you are learning about technology, there always this strange catch at the back of your head: When will this piece of knowledge go outdated?

Technology moves at an incredible pace. What you learn today will eventually become obselete in a couple of years. In some cases, months.

As a teacher myself, I never provide my students answers to the questions they have. Only pointing them at the right directions for them to seek the answer. In some cases, I ended up learning along side with them.

You see, the skill of acquiring knowledge is more important than the knowledge itself. Knowledge is an object; you can get it, you can lose it. But the skill to acquire knowledge stays with you till the end.

When it comes to technology, such a skill is arguably the most important skill of all. It is the skill that allows you to stay relevant in the midst of a fast changing environment.