Strange memory hog in Ubuntu 16.04

Written on Sun, 3 July 2016

Ever since I booted up my Ubuntu 16.04 on AWS EC2 T2.nano to host this site, I often have a problem with kswapd0. It's a very strange occurance. I've been investigating this issue for many weeks. Each time, I come back to not much information to resolve this permanently.

Whenever I check my memory with this command

top

I see kswapd0 utilising almost fully all of the CPU cycle. As you know, t2.nano is given limited amount of CPU credits to use. This CPU cycle spike that doesn't resolve will consume all of it within a few minutes.

If you're experiencing this issue too, the recommended solution is of course to migrate to a better instance. But if you're like me, already bought down a reserve instance and refuse to be defeated by some strange programming logic in Linux, here's some of the things you can do.

Clear the cache every interval

*/5 * * * * echo 3 > /proc/sys/vm/drop_caches

Use this piece of code in your cronjobs. This will force the cache to clear every 5 minutes. This is the number 1 thing that you need to do. It's not 100% solution but it will almost immediate bring down the occurance of this CPU spike to once a month or so.

When the CPU cycle spikes, do the next step.

Check which application is consuming a chunk of memory

top

This command won't help to identify which application is using memory. Use this instead:

htop

You might need to install it:

apt-get install htop

In the interface, you need to sort out the application using the most memory. In my case, it was mysql that was consuming a lot of memory. What I needed to do was just restart mysql and the CPU cycle goes back to normal. In some cases, you might need to terminate the application. Do so with caution.

Update your linux installation

This isn't exactly a solution. Since I don't know the exact cause of this problem, perhaps let's keep Ubuntu and the relevant applications constantly updated with the latest bug fixes. Who knows the makers of the application spotted this problem and issued a fix.

apt-get update
apt-get dist-upgrade
apt-get autoremove

Usually I would do a reboot after this installation, just to be extra sure that the newest codes are running. Given that we have no clue why this problem occurs, being on the safe side pays.

Reboot the computer

You have been clearing the cache. You tried to identify the offending application and resolve it. kswapd0 is still eating away all your precious CPU cycle.

You are not alone. This is your last resort. Reboot the computer with:

reboot

Then you make sure all your services are back up and running.

Permanent solution needed

If there is anybody who has a clue what causes this strange CPU spike, please do share.