Swap Partition on EC2 Ubuntu

Written on Wed, 15 February 2017

If you have a case of having file system as btrfs, you will know that you can't have swap files on such systems. In such case, you probably need a partition. If you're running your server on AWS EC2, then you can even opt for having SSD as your swap partition, especially if your other EBS if on HDD.

Step 1: Create your EBS and attach it to your EC2 instance

You will have to create a new volume in your AWS console or via AWS API.

• Volume Type: GP2 or IO1
  • You can usually settle for GP2, but if you need the extra performance boost, you can go IO1. If you still need better performance, you should just launch an EC2 instance that has more RAM. If you don't mind a performance hit, you can go for st1 or sc1.
• Size: 8GB
  • More capacity if you find yourself needing more. Although keep in mind that if you need more than 2 times the size of your RAM, you might want to consider a better EC2 instance with more RAM instead. I'm going to assume in this tutorial that you are using 8GB RAM.
• Availability Zone: The one which your instance resides in

Once you have created your volume, you must attach it to your instance.

Step 2: Find your newly attached volume

You can find your newly attached volume with:

sudo lsblk

You should see a disk called xvdf, or another letter, that has 8G of space. In this tutorial, I'm assuming your disk is xvdf.

Step 3: Check for existing swap

In this tutorial, I'm assuming you have no swap.

You can check for existing swap with:

sudo swapon --show

If there is no output, that means you have no swap space currently.

You can verify no existing swap with:

free -h

If you see that swap has 0B, that means there is no active swap present in system.

Step 4: Making swap

Format your partition with:

sudo mkswap /dev/xvdf

You should see something like:

Setting up swapspace version 1, size = 8 GiB (8589930496 bytes)
no label, UUID=d3a23c79-5144-47fa-b422-885ebf63e2fc

Take note of your UUID. You will need it.

Step 5: Enabling swap on boot

You need to configure your fstab with the new swap partition.

echo 'UUID=d3a23c79-5144-47fa-b422-885ebf63e2fc none swap sw 0 0' | sudo tee -a /etc/fstab

Be sure to replace your UUID in this command.

Reboot your computer for swap to take effect.

reboot

Step 6: Check your new swap space

Run these commands to see that you have 8G for swap space.

sudo swapon --show
free -h

If you don't see your new swap space, something has gone wrong somewhere as you were following this tutorial.

Step 7: Configuring swappiness (Optional)

Swappiness is a parameter for the system to determine how often your system swaps data out of RAM to swap space. The values are represented from 0 to 100, in percentages. By default, the value is 60.

You must remember that a swap on HDD is time costly, while a swap in SSD is lifespan costly. Since I'm running a server, I would prefer a value closer to 0, only to use RAM when absolutely neccessary. In some use cases, using swap actually increases performance.

Choose your ideal percentage, then run the following:

sudo sysctl vm.swappiness=10

I chose 10 so that it will use swap sparingly.

To make this value permanent during reboots, open /etc/sysctl.conf:

sudo nano /etc/sysctl.conf

Then add the following at the bottom:

vm.swappiness=10

Conclusion

Having swap has many advantages. In some cases, it will safe a technician from certain doom due to application crashes.

For me, I managed to lower my operating cost of running instances in AWS. After all, if I don't mind a bit of performace hit from running swap in SSD. In return, I get to half the cost of the instance running cost.

If you're looking to have a swap file instead of a swap partition, check out my other tutorial.

Swap Space in EC2 Ubuntu

Written on Mon, 13 February 2017

Over the course of running this webserver and zimbra, I realised that there is a need to have swap space whenever you have applications that are memory hungry. My webserver has occasionally run out of memory on t2.nano instance and my zimbra webserver frequently has memory issues, almost always due to MySQL/MariaDB. MySQL/MariaDB certainly is memory hungry for some reasons, so I guess until I can figure out how to optimise these applications, I just have to work with either increasing RAM or having swap space.

The Ubuntu instances will likely not have swap space. You must configure one. In this tutorial, I will make things really easy by making a swapfile in the filesystem instead of creating a partition

Before you proceed with the tutorial, you must take into consideration of several matters. If you EBS volume is Magnetic, sc1, or st1, having a swap will reduce the performance of applications. Spinning disk will always be slower than RAM or SSD. However, if you are gp2 or io1, then you will see noticable performance of having swap on SSD than on HDD. But you will degrade the SSD fast due to many write cycles. Either way, having swap is a very good safeguard and comparatively affordable solution against application error or crashes related to out-of-memory situations.

Incompatibilities

You must also take note if your file system supports swap files. At the time of this writing, btrfs is not compatible with swap. You probably want to create a partition instead.

Step 1: Checking for existing Swap

We can see if the system has any configured swap by typing:

sudo swapon --show

If there is no output, that means you have no swap space currently.

You can verify no existing swap with:

free -h

If you see that swap has 0B, that means there is no active swap present in system.

Step 2: Check available space in file system

We check the available space in the filesystem with:

df -h

Take note of the space space available in /dev/xvda1. If you intend to have swap space on other EBS, then check the available space of it.

Planning your swap space

In a standard Ubuntu installation, you should have 8GB hard disk space, of which, the OS and the applications will consume some. You would probably be left with about 5GB. It is usually recommended to have twice as much swap as your RAM. Of course, you are free to have as much swap as you need. In a t2.nano instances, you will only have 512MB RAM. so I'm going to have 1GB of swap space.

Step 3: Creating Swap File

I'm going to create 1GB of swap space:

sudo fallocate -l 1G /swapfile

We can verify that the correct amount of space was reserved with:

ls -lh /swapfile

You should see something like:

-rw-r--r-- 1 root root 1.0G Feb 13 07:35 /swapfile

Step 4: Enabling swap space

For security reasons, we should only allow users with root privileges to read the contents. All other users should not have access to the swap file.

sudo chmod 600 /swapfile

Verify the permissions change with:

ls -lh /swapfile

You should see something like:

-rw------- 1 root root 1.0G Feb 13 07:35 /swapfile

We now mark the file as swap space with:

sudo mkswap /swapfile

You should see an output like:

Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
no label, UUID=0d7257f1-9def-4542-bcba-34b8f50f2822

Now we enable swap:

sudo swapon /swapfile

Verify the swap is enabled with:

sudo swapon --show

You will now see a swap file with 1024M.

You can also check with:

free -h

You will see 1.0G of space for use.

Step 5: Making Swap Permanent

If you reboot your computer, the setting you just did will be gone. You can make swap a permanent feature during reboot.

Add swap file information to fstab with:

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

That's it!

Step 6: Configuring swappiness (Optional)

Swappiness is a parameter for the system to determine how often your system swaps data out of RAM to swap space. The values are represented from 0 to 100, in percentages. By default, the value is 60.

You must remember that a swap on HDD is time costly, while a swap in SSD is lifespan costly. Since I'm running a server, I would prefer a value closer to 0, only to use RAM when absolutely neccessary. In some use cases, using swap actually increases performance.

Choose your ideal percentage, then run the following:

sudo sysctl vm.swappiness=10

I chose 10 so that it will use swap sparingly.

To make this value permanent during reboots, open /etc/sysctl.conf:

sudo nano /etc/sysctl.conf

Then add the following at the bottom:

vm.swappiness=10

Conclusion

Swap has many many advantages. In some cases, it will save a technician from certain doom due to application crashes.

Generally swap is the most affordable alternative to getting RAM upgrades, or even CPU upgrades. Use it wisely and you will have a well-oiled affordable machine that will last you very long.

If you have to create a swap partition instead of a swap file, check out my other tutorial on creating a swap partition.

Decrypting JSON from XE Currency

Written on Thu, 12 January 2017

Disclaimer: What I'm about to show you is against XE Currency's Terms of Use. Use the follow codes at your discretion.

So I have been exploring ways to obtain forex data from external sources. But many websites offer paid services. Me being just a lone developer and not able to afford it, I decided to just scrape of the data from websites. That's where I hit a stumbling block with XE Currency.

An example of the URL to get XE Currency historic rates is http://www.xe.com/currencycharts/currates.php?from=USD&to=SGD&amount=1&_=1484224006201. You will see that this isn't JSON at all. Something is amissed. It is using this URL, the javascript interprets it as JSON, yet this looks nothing like JSON.

So after much delving on this matter, I realised, not to my surprise, that it is encrypted. Interestingly, they do not encrypt the entire JSON, only every 10th character of the JSON string. The rest of the string is just obfuscated. Both the encryption and the obfuscation looks very similar to caesar cipher.

So if you are going to scrape the data off their website, I'll show you the codes I used. I doubt there will be any legal issues for me posting this, since their entire decryption algorithm is publicly accessible. I merely took their algorithm and implement it in PHP.

//Once you get the encrypted JSON, put it here.
$encryptedJSON = "ENCRYPTEDJSONHERE";

//Trim any whitespaces
$encryptedJSON = trim($encryptedJSON);

//The decryption key is the last 4 characters of the encrypted JSON
$hiddenKey = substr($encryptedJSON, strlen($encryptedJSON) - 4);

$decryptedKey = ord(substr($hiddenKey, 0)) + ord(substr($hiddenKey, 1)) + ord(substr($hiddenKey, 2)) + ord(substr($hiddenKey, 3));
$decryptedKey = (strlen($encryptedJSON) - 10) % $decryptedKey;
$decryptedKey = ($decryptedKey > strlen($encryptedJSON) - 10 - 4) ? (strlen($encryptedJSON) - 10 - 4) : $decryptedKey;

//The actual decryption key is hidden in the middle of the JSON
$decryptedKey2 = substr($encryptedJSON, $decryptedKey, 10);

//Remove the encryption key from the JSON string
$encryptedJSON = substr($encryptedJSON, 0, $decryptedKey).substr($encryptedJSON, $decryptedKey + 10);

//Decode URI
$encryptedJSON = urldecode($encryptedJSON);

//Character shift process, doesn't involve any key
$stringList = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";
$cursor = 0;
$shiftedJSON = "";

//Clear Unwanted characters
$encryptedJSON = preg_replace("/[^A-Za-z0-9\+\/\=]/m", "", $encryptedJSON);

do {
$char1 = strpos($stringList, substr($encryptedJSON, $cursor++, 1));
$char2 = strpos($stringList, substr($encryptedJSON, $cursor++, 1));
$char3 = strpos($stringList, substr($encryptedJSON, $cursor++, 1));
$char4 = strpos($stringList, substr($encryptedJSON, $cursor++, 1));

$code1 = ($char1 << 2) | ($char2 >> 4);
$code2 = (($char2 & 15) << 4) | ($char3 >> 2);
$code3 = (($char3 & 3) << 6) | $char4;

$shiftedJSON .= chr($code1);
if ($char3 != 64) {
$shiftedJSON .= chr($code2);
}
if ($char4 != 64) {
$shiftedJSON .= chr($code3);
}
} while ($cursor < strlen($encryptedJSON));
$encryptedJSON = urldecode($shiftedJSON);

//Decrypt every 10th character
$counter = 0;
$decryptedJSON = "";
for ($counter10th = 0; $counter10th < strlen($encryptedJSON); $counter10th+=10) {
$encryptedChar = $encryptedJSON[$counter10th];
$shiftKey = $decryptedKey2[($counter % strlen($decryptedKey2) - 1) < 0 ? (strlen($decryptedKey2) + ($counter % strlen($decryptedKey2) - 1)) : ($counter % strlen($decryptedKey2) - 1)];
$encryptedChar = chr(ord($encryptedChar) - ord($shiftKey));
$decryptedJSON .= $encryptedChar.substr($encryptedJSON, $counter10th + 1, 9);
$counter++;
}

//There you go! The decrypted JSON.
echo json_decode($decryptedJSON);

A reminder to read my disclaimer!

Controlling Docker Containers with PHP

Written on Fri, 6 January 2017

After searching through the internet to find out how to manage docker containers through PHP, I finally figured out how to do it. I assume you are using Ubuntu 16.04 and the docker you are using is the latest to date.

Add www-data to docker

According to the documentation, the docker daemon always runs as the root user. Since Docker version 0.5.2, the docker daemon binds to a Unix socket instead of a TCP port. By default that Unix socket is owned by the user root, and so, by default, you can access it with sudo.

Starting in version 0.5.3, if you (or your Docker installer) create a Unix group called docker and add users to it, then the docker daemon will make the ownership of the Unix socket read/writable by the docker group when the daemon starts. The docker daemon must always run as the root user, but if you run the docker client as a user in the docker group then you don't need to add sudo to all the client commands.

So we are going to add www-data to docker so that it can run docker commands. First, let's create the group if it doesn't exist. Skip this command if the group exists.

sudo groupadd docker

Add www-data to docker group.

sudo gpasswd -a www-data docker

Reboot the computer for all the permissions to take effect.

sudo reboot

Writing docker commands

In PHP, you will need to specify the commands in this syntax:

shell_exec("RET=`docker command `;echo $RET');

For example, if you want to run hello world, you write this:

shell_exec('RET=`docker run hello-world`;echo $RET');

Build yourself a Comprehensive Business Suite with Odoo

Written on Fri, 30 December 2016

Odoo is an impressive Business Suite that contains so many enterprise management tools. The best part of Odoo is that most of the features are free. There are only a few paid features that would be very useful if you are managing more than 50 employees in your company. Of course, it is up to your use case to see if the enterprise features are useful. In this tutorial, I'll show you how to set up your own Odoo 10 Community Edition, which you can freely use if you are running a small company with very little employees.

---

Setup Odoo 10

Installing Odoo 10 on Ubuntu 16.04 is easy. Much of this tutorial is taken from the documentation. Feel free to review the documentation when in doubt.

Keep Ubuntu updated with the latest packages

apt-get update && apt-get -y dist-upgrade && reboot

Reboot your server to ensure everything is loaded with the latest packages.

Let's start with adding the repository for Odoo 10, then installing it.

wget -O - https://nightly.odoo.com/odoo.key | apt-key add -
echo "deb http://nightly.odoo.com/10.0/nightly/deb/ ./" >> /etc/apt/sources.list
apt-get update && apt-get install -y odoo

The configuration can be found in /etc/odoo/odoo.conf

nano /etc/odoo/odoo.conf

If you wish to set things to default, just leave the file as it is.

---

Running Odoo on port 80

If you're like me, I don't like to type the port number in the URL. And my server is running just 1 application. So why not use port 80?

nano /etc/rc.local

Add the following before exit 0 command.

iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8069
iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 8070
iptables-save

Then reboot the server

reboot

Browse to the server with your preferred browser, such as http://localhost if you have changed port, or http://localhost:8069 if you left it as default. From here on, it's straight forward installing modules and applications for your company needs. More information as Odoo's user guide.

---

If you're a developer looking to develop for Odoo 10, I recommend reading Odoo 10 Development Essentials. Companies who hire Odoo developers are usually looking to streamline their business to be more efficient. Many big companies will want to do that to reduce operational cost.