The 4 levels of IT Architecture

Written on Thu, 9 March 2017

20 years ago, developing web applications was just a simple straight forward thing. Today, developing web applications is a multi-disciplinary work. I’m going to break down the levels of work while keeping things as simplified as possible.

Think of a web application as a building with 4 levels:

Each level rest on the level before it. You can’t have level 2 without level 1, level 3 without level 2, and level 4 without level 3. The same reasoning that you can’t build a building without the storey before it.

In the simplest of web applications, you can build all 4 levels into a single project. The most complex applications can even have multiple projects in each level.

Level 1: Matrix

The central node to everything else. The brains of everything. The business logic. The data storage platform.

Here are the things you need to consider in this level, but not limited to:

• OS: Linux, Windows, Unix, etc?
• Single server or clustered environment?
• RDBS: MySQL, Oracle, etc?
• NoSQL or not?
• Assets: Filesystem, NAS, S3, etc?
• Backups?
• Data Cache: Memcache, file cache, etc?
• All the required business logic, such as authentication, access control list, machine learning, etc
• Obviously, programming language: PHP, Java, .NET, etc?

Level 2: Connector

A connector is a bridge between the Matrix and the interface. In the simplest of web application, sometimes, this is non-existent. You will only need a connector if your application has a distinct seperation between the Matrix and the interface.

A mail server is an example of a connector. When you develop a web application, you are certainly not going to build your own mail server along with it. You will likely use an external provider, such as Gmail, Hotmail, Yahoo, etc… Your connector will send out emails on behalf of your application.

A connector can also be an API from which other applications to interact with. For example, Facebook API is a connector to Facebook itself. The API allows you to interact with Facebook. So if you are building your own API, you are allowing other applications to interact with your own web application. You will most likely need this if you are building a mobile application.

Here are the things you need to consider in this level, but not limited to:

• What protocol should I use for API, JSON, XML, or binary, for other applications to interact with my application?
• Are there external services that I require to connect with? Such as mail server, Facebook API, etc…
• Do I need authentication to connect to other services? Do I need other applications to authenticate with my application before interacting with it?
• How long will DNS propagation take?

Level 3: Interface

The interface which a user will use to interact with the Matrix through the connecter, such as the web browser, the phone, the tablet, the Xbox, Outlook, Hololens, Oculus, etc…

In your context, I know you explicitly mention about web applications. But web applications can be consumed on mobile phones as well. I also presume that your application has a chance to go beyond a website.

Here are the things you need to consider in this level, but not limited to:

• Web Programming language: Do I use server side scripting language such as PHP, Java? Or do I use client side scripting language, such as Javascript, AngularJS, ReactJS, etc…?
• Device Programming language: If I’m building a mobile application, should I use Java, C#, etc..?
• Mail Client: Text, HTML, or both?
• Background services: Push Notifications for Mobile application? Polling services for updates?
• Device functions: Location Detection? Accelerometer? Camera?

Level 4: Reality

The presentation and the delivery of content, inclusive of text, images, videos, live broadcast, etc…

Here is where the users actually see, touch, and feel your application. It is not restricted to a human user. A robot, such as googlebot, can also be viewing your website on this level. So other than making sure that your website is readable to a human, you would also consider Search Engine Optimisation (SEO).

Here are the things you need to consider in this level, but not limited to:

• Presentation Language: HTML, CSS, Javascript, Flash, etc…
• External loaded files: Javascript libraries, images, videos, documents, etc…
• Sementic Language for robots, such as GoogleBot, OpenGraph crawler, etc…
• Input source: Mouse? Touch Screen? Joystick? Keyboard? Kinect?
• Output Device: Monitor screen? Mega Stadium Multimedia systems? 5.1 Surround sound? Headphones? Augmented reality through Hololens? Mobile screens?
• Web Browser: To which version of browser do I support till for IE, Edge, chrome, FireFox, Safari, and Opera?
• Resolution: Tiny mobile screens to mega displays in concerts?

Typical Simple Use Case

In a typical use case, someone such as yourself, will probably rent a web host from GoDaddy(Maybe). And then from its cPanel, you will instruct it to install wordpress. And that’s it for Level 1.

Now you need to send out emails to your subscribers, so you also get a mail server from GoDaddy(Maybe) and you configure your wordpress to send out emails through GoDaddy SMTP. And that’s it for Level 2.

Now you want to have a nice looking website that can be viewed nicely on mobile phone browsers too. So you look around the internet and downloaded a nice looking template for wordpress. And that’s it for Level 3 and 4.

A (somewhat) Complex Use Case

Level 1
You decided to rent servers from AWS. In Level 1, you decided that the Matrix is going to be in a clustered environment. So you must store Relational Data in a RDS. Your assets must also be stored in S3. You have configure VPN site-to-site configuration, so all data backups go to your office servers. You use Java to build the matrix, but you also use machine learning in your application, which is built on python. As machine learning is computationally intensive, you built a seperate clustered GPU system for your ML needs.

Level 2
Not just a website, you also intend to have mobile applications, PlayStation, Xbox, Hololens. Your web application will have a built in connector to do HTTP calls to the matrix in JSON. You will need to build a seperate connector for your mobile applications, Playstation, Xbox, Hololens. You anticipate that you have high traffic volume, so you add a load balancer to your connectors. Your mail server is Zimbra, which you are hosting on AWS. So you ensure that emails are sent out properly. You also have SMS notifications, so you subscribe to a SMS gateway.

Level 3
As you are building for web browsers, mobiles, PlayStation, Xbox, and Hololens, you build all of these applications in their respective programming language.As you are also using Facebook Login, Google+ Login, and Paypal Login, you have to build these services into every application. You also ensure your emails do not fall into spam folder when it is delivered. On mobile devices, you are accessing camera and location functions. All of these external devices are communicating with Level 2 in either Binary or JSON.

Level 4
You have to take into account of the various mode of input and output, ensuring the best user experience in every device. Due to high volume traffic, you decide to have a Content Distribution Network to lower latency. Robots are also reading your contents, so SEO must be properly done.

Swap Partition on EC2 Ubuntu

Written on Wed, 15 February 2017

If you have a case of having file system as btrfs, you will know that you can't have swap files on such systems. In such case, you probably need a partition. If you're running your server on AWS EC2, then you can even opt for having SSD as your swap partition, especially if your other EBS if on HDD.

Step 1: Create your EBS and attach it to your EC2 instance

You will have to create a new volume in your AWS console or via AWS API.

• Volume Type: GP2 or IO1
  • You can usually settle for GP2, but if you need the extra performance boost, you can go IO1. If you still need better performance, you should just launch an EC2 instance that has more RAM. If you don't mind a performance hit, you can go for st1 or sc1.
• Size: 8GB
  • More capacity if you find yourself needing more. Although keep in mind that if you need more than 2 times the size of your RAM, you might want to consider a better EC2 instance with more RAM instead. I'm going to assume in this tutorial that you are using 8GB RAM.
• Availability Zone: The one which your instance resides in

Once you have created your volume, you must attach it to your instance.

Step 2: Find your newly attached volume

You can find your newly attached volume with:

sudo lsblk

You should see a disk called xvdf, or another letter, that has 8G of space. In this tutorial, I'm assuming your disk is xvdf.

Step 3: Check for existing swap

In this tutorial, I'm assuming you have no swap.

You can check for existing swap with:

sudo swapon --show

If there is no output, that means you have no swap space currently.

You can verify no existing swap with:

free -h

If you see that swap has 0B, that means there is no active swap present in system.

Step 4: Making swap

Format your partition with:

sudo mkswap /dev/xvdf

You should see something like:

Setting up swapspace version 1, size = 8 GiB (8589930496 bytes)
no label, UUID=d3a23c79-5144-47fa-b422-885ebf63e2fc

Take note of your UUID. You will need it.

Step 5: Enabling swap on boot

You need to configure your fstab with the new swap partition.

echo 'UUID=d3a23c79-5144-47fa-b422-885ebf63e2fc none swap sw 0 0' | sudo tee -a /etc/fstab

Be sure to replace your UUID in this command.

Reboot your computer for swap to take effect.

reboot

Step 6: Check your new swap space

Run these commands to see that you have 8G for swap space.

sudo swapon --show
free -h

If you don't see your new swap space, something has gone wrong somewhere as you were following this tutorial.

Step 7: Configuring swappiness (Optional)

Swappiness is a parameter for the system to determine how often your system swaps data out of RAM to swap space. The values are represented from 0 to 100, in percentages. By default, the value is 60.

You must remember that a swap on HDD is time costly, while a swap in SSD is lifespan costly. Since I'm running a server, I would prefer a value closer to 0, only to use RAM when absolutely neccessary. In some use cases, using swap actually increases performance.

Choose your ideal percentage, then run the following:

sudo sysctl vm.swappiness=10

I chose 10 so that it will use swap sparingly.

To make this value permanent during reboots, open /etc/sysctl.conf:

sudo nano /etc/sysctl.conf

Then add the following at the bottom:

vm.swappiness=10

Conclusion

Having swap has many advantages. In some cases, it will safe a technician from certain doom due to application crashes.

For me, I managed to lower my operating cost of running instances in AWS. After all, if I don't mind a bit of performace hit from running swap in SSD. In return, I get to half the cost of the instance running cost.

If you're looking to have a swap file instead of a swap partition, check out my other tutorial.

Swap Space in EC2 Ubuntu

Written on Mon, 13 February 2017

Over the course of running this webserver and zimbra, I realised that there is a need to have swap space whenever you have applications that are memory hungry. My webserver has occasionally run out of memory on t2.nano instance and my zimbra webserver frequently has memory issues, almost always due to MySQL/MariaDB. MySQL/MariaDB certainly is memory hungry for some reasons, so I guess until I can figure out how to optimise these applications, I just have to work with either increasing RAM or having swap space.

The Ubuntu instances will likely not have swap space. You must configure one. In this tutorial, I will make things really easy by making a swapfile in the filesystem instead of creating a partition

Before you proceed with the tutorial, you must take into consideration of several matters. If you EBS volume is Magnetic, sc1, or st1, having a swap will reduce the performance of applications. Spinning disk will always be slower than RAM or SSD. However, if you are gp2 or io1, then you will see noticable performance of having swap on SSD than on HDD. But you will degrade the SSD fast due to many write cycles. Either way, having swap is a very good safeguard and comparatively affordable solution against application error or crashes related to out-of-memory situations.

Incompatibilities

You must also take note if your file system supports swap files. At the time of this writing, btrfs is not compatible with swap. You probably want to create a partition instead.

Step 1: Checking for existing Swap

We can see if the system has any configured swap by typing:

sudo swapon --show

If there is no output, that means you have no swap space currently.

You can verify no existing swap with:

free -h

If you see that swap has 0B, that means there is no active swap present in system.

Step 2: Check available space in file system

We check the available space in the filesystem with:

df -h

Take note of the space space available in /dev/xvda1. If you intend to have swap space on other EBS, then check the available space of it.

Planning your swap space

In a standard Ubuntu installation, you should have 8GB hard disk space, of which, the OS and the applications will consume some. You would probably be left with about 5GB. It is usually recommended to have twice as much swap as your RAM. Of course, you are free to have as much swap as you need. In a t2.nano instances, you will only have 512MB RAM. so I'm going to have 1GB of swap space.

Step 3: Creating Swap File

I'm going to create 1GB of swap space:

sudo fallocate -l 1G /swapfile

We can verify that the correct amount of space was reserved with:

ls -lh /swapfile

You should see something like:

-rw-r--r-- 1 root root 1.0G Feb 13 07:35 /swapfile

Step 4: Enabling swap space

For security reasons, we should only allow users with root privileges to read the contents. All other users should not have access to the swap file.

sudo chmod 600 /swapfile

Verify the permissions change with:

ls -lh /swapfile

You should see something like:

-rw------- 1 root root 1.0G Feb 13 07:35 /swapfile

We now mark the file as swap space with:

sudo mkswap /swapfile

You should see an output like:

Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
no label, UUID=0d7257f1-9def-4542-bcba-34b8f50f2822

Now we enable swap:

sudo swapon /swapfile

Verify the swap is enabled with:

sudo swapon --show

You will now see a swap file with 1024M.

You can also check with:

free -h

You will see 1.0G of space for use.

Step 5: Making Swap Permanent

If you reboot your computer, the setting you just did will be gone. You can make swap a permanent feature during reboot.

Add swap file information to fstab with:

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

That's it!

Step 6: Configuring swappiness (Optional)

Swappiness is a parameter for the system to determine how often your system swaps data out of RAM to swap space. The values are represented from 0 to 100, in percentages. By default, the value is 60.

You must remember that a swap on HDD is time costly, while a swap in SSD is lifespan costly. Since I'm running a server, I would prefer a value closer to 0, only to use RAM when absolutely neccessary. In some use cases, using swap actually increases performance.

Choose your ideal percentage, then run the following:

sudo sysctl vm.swappiness=10

I chose 10 so that it will use swap sparingly.

To make this value permanent during reboots, open /etc/sysctl.conf:

sudo nano /etc/sysctl.conf

Then add the following at the bottom:

vm.swappiness=10

Conclusion

Swap has many many advantages. In some cases, it will save a technician from certain doom due to application crashes.

Generally swap is the most affordable alternative to getting RAM upgrades, or even CPU upgrades. Use it wisely and you will have a well-oiled affordable machine that will last you very long.

If you have to create a swap partition instead of a swap file, check out my other tutorial on creating a swap partition.

Decrypting JSON from XE Currency

Written on Thu, 12 January 2017

Disclaimer: What I'm about to show you is against XE Currency's Terms of Use. Use the follow codes at your discretion.

So I have been exploring ways to obtain forex data from external sources. But many websites offer paid services. Me being just a lone developer and not able to afford it, I decided to just scrape of the data from websites. That's where I hit a stumbling block with XE Currency.

An example of the URL to get XE Currency historic rates is http://www.xe.com/currencycharts/currates.php?from=USD&to=SGD&amount=1&_=1484224006201. You will see that this isn't JSON at all. Something is amissed. It is using this URL, the javascript interprets it as JSON, yet this looks nothing like JSON.

So after much delving on this matter, I realised, not to my surprise, that it is encrypted. Interestingly, they do not encrypt the entire JSON, only every 10th character of the JSON string. The rest of the string is just obfuscated. Both the encryption and the obfuscation looks very similar to caesar cipher.

So if you are going to scrape the data off their website, I'll show you the codes I used. I doubt there will be any legal issues for me posting this, since their entire decryption algorithm is publicly accessible. I merely took their algorithm and implement it in PHP.

//Once you get the encrypted JSON, put it here.
$encryptedJSON = "ENCRYPTEDJSONHERE";

//Trim any whitespaces
$encryptedJSON = trim($encryptedJSON);

//The decryption key is the last 4 characters of the encrypted JSON
$hiddenKey = substr($encryptedJSON, strlen($encryptedJSON) - 4);

$decryptedKey = ord(substr($hiddenKey, 0)) + ord(substr($hiddenKey, 1)) + ord(substr($hiddenKey, 2)) + ord(substr($hiddenKey, 3));
$decryptedKey = (strlen($encryptedJSON) - 10) % $decryptedKey;
$decryptedKey = ($decryptedKey > strlen($encryptedJSON) - 10 - 4) ? (strlen($encryptedJSON) - 10 - 4) : $decryptedKey;

//The actual decryption key is hidden in the middle of the JSON
$decryptedKey2 = substr($encryptedJSON, $decryptedKey, 10);

//Remove the encryption key from the JSON string
$encryptedJSON = substr($encryptedJSON, 0, $decryptedKey).substr($encryptedJSON, $decryptedKey + 10);

//Decode URI
$encryptedJSON = urldecode($encryptedJSON);

//Character shift process, doesn't involve any key
$stringList = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";
$cursor = 0;
$shiftedJSON = "";

//Clear Unwanted characters
$encryptedJSON = preg_replace("/[^A-Za-z0-9\+\/\=]/m", "", $encryptedJSON);

do {
$char1 = strpos($stringList, substr($encryptedJSON, $cursor++, 1));
$char2 = strpos($stringList, substr($encryptedJSON, $cursor++, 1));
$char3 = strpos($stringList, substr($encryptedJSON, $cursor++, 1));
$char4 = strpos($stringList, substr($encryptedJSON, $cursor++, 1));

$code1 = ($char1 << 2) | ($char2 >> 4);
$code2 = (($char2 & 15) << 4) | ($char3 >> 2);
$code3 = (($char3 & 3) << 6) | $char4;

$shiftedJSON .= chr($code1);
if ($char3 != 64) {
$shiftedJSON .= chr($code2);
}
if ($char4 != 64) {
$shiftedJSON .= chr($code3);
}
} while ($cursor < strlen($encryptedJSON));
$encryptedJSON = urldecode($shiftedJSON);

//Decrypt every 10th character
$counter = 0;
$decryptedJSON = "";
for ($counter10th = 0; $counter10th < strlen($encryptedJSON); $counter10th+=10) {
$encryptedChar = $encryptedJSON[$counter10th];
$shiftKey = $decryptedKey2[($counter % strlen($decryptedKey2) - 1) < 0 ? (strlen($decryptedKey2) + ($counter % strlen($decryptedKey2) - 1)) : ($counter % strlen($decryptedKey2) - 1)];
$encryptedChar = chr(ord($encryptedChar) - ord($shiftKey));
$decryptedJSON .= $encryptedChar.substr($encryptedJSON, $counter10th + 1, 9);
$counter++;
}

//There you go! The decrypted JSON.
echo json_decode($decryptedJSON);

A reminder to read my disclaimer!

Controlling Docker Containers with PHP

Written on Fri, 6 January 2017

After searching through the internet to find out how to manage docker containers through PHP, I finally figured out how to do it. I assume you are using Ubuntu 16.04 and the docker you are using is the latest to date.

Add www-data to docker

According to the documentation, the docker daemon always runs as the root user. Since Docker version 0.5.2, the docker daemon binds to a Unix socket instead of a TCP port. By default that Unix socket is owned by the user root, and so, by default, you can access it with sudo.

Starting in version 0.5.3, if you (or your Docker installer) create a Unix group called docker and add users to it, then the docker daemon will make the ownership of the Unix socket read/writable by the docker group when the daemon starts. The docker daemon must always run as the root user, but if you run the docker client as a user in the docker group then you don't need to add sudo to all the client commands.

So we are going to add www-data to docker so that it can run docker commands. First, let's create the group if it doesn't exist. Skip this command if the group exists.

sudo groupadd docker

Add www-data to docker group.

sudo gpasswd -a www-data docker

Reboot the computer for all the permissions to take effect.

sudo reboot

Writing docker commands

In PHP, you will need to specify the commands in this syntax:

shell_exec("RET=`docker command `;echo $RET');

For example, if you want to run hello world, you write this:

shell_exec('RET=`docker run hello-world`;echo $RET');