Persistent RAM disk – extending SD card lifetime

Internet forums are full of stories about SD card corruption and data loss when running Raspberry PI. It happened to me as well and obviously I wasn't very happy about that. It took me a while to recover all the data from backups and make the PI running again. I decided to find the way how to avoid this problem in the future or at least reduce the chances of card being corrupted.

The overall problem is called SD card wear out. The main point is flash memory media have erasable cells, each of which can be put through a limited number of erase cycles before becoming faulty. Usually it takes thousands of cycles which may look like a huge number, but it depends what are you doing with you Raspberry. If your app is not designed carefully or worse you're using persistent database, thousands of erase cycles can be depleted within few days.

I came across this problem in discussion about Monitoring internet connection with Smokeping. Smokeping uses RRDTool as internal database for measured data. And since the monitoring is continuous process it writes a new data to SD card every few minutes. That can contribute to SD card wear out pretty significantly.

It's crucial to reduce write operations to SD card. The obvious solution is to use RAM disk or tmpfs (i. e. file system that stores all the files in volatile memory instead of persistent storage). But problems will arise when you realize you'll lose all the data during reboot. For example if we move /var/log to tmpfs we'll always start with empty directory after reboot. That won't work very well since certain apps expect some specific directory structure within /var/log and they'll crash at startup.

More advanced solution is to combine tmpfs with persistent storage effectively creating persistent ramdisk. So the life-cycle of data will look like this:

• After boot data from chosen directory is copied from SD card to tmpfs
• When data is modified it happens at no cost in RAM
• To prevent data loss during electricity outage or random restart there is hourly back-up made to SD card
• When properly shutting down all actual data from tmpfs are copied back to SD card

Anything-sync-daemon

I found a great tool called Anything-sync-daemon written in bash that does exactly what we need – maps chosen directories to tmpfs and syncs changes back to SD card when needed.

Unfortunately Anything-sync-daemon is available only for Arch linux so I decided to compile and build my own package for Debian/Raspbian 8 Jessie.

Anything-sync-daemon package for Debian

All commands need to be run as root, thus sudo su

Install required dependencies for building a package and rsync

apt-get install git build-essential checkinstall rsync 

Clone the current source code of Anything-sync-daemon from Github

git clone https://github.com/graysky2/anything-sync-daemon.git

Open the source code directory and checkout the current version

cd anything-sync-daemon/ && git checkout v5.81

Build the Debian package and install it

make && checkinstall --pkgversion 5.81 make install-systemd-all

Now it's time to setup directories that need to be managed by asd. Edit /etc/asd.conf

WHATTOSYNC=('/var/lib/smokeping' '/var/log')
VOLATILE="/dev/shm"

• WHATTOSYNC – List of directories that are mapped to RAM
• VOLATILE – Path of the currently mounted volatile memory (tmpfs)

And finally restart the Anything-sync-daemon and enable it to start after boot.

systemctl enable asd.service && service asd restart

Debugging Anything-sync-daemon

There is an option to print the current configuration and state of monitored directories

# asd p

Anything-sync-daemon v5.81 on Raspbian GNU/Linux 8 (jessie)

 Systemd service is currently active.
 Systemd resync service is currently active.
 Overlayfs technology is currently inactive.

Asd will manage the following per /run/asd.conf settings:

 owner/group id:     smokeping/115
 target to manage:   /var/lib/smokeping
 sync target:        /var/lib/.smokeping-backup_asd
 tmpfs target:       /dev/shm/asd-smokeping/var/lib/smokeping
 dir size:           12M
 recovery dirs:      none

Useful links

• Anything-sync-daemon homepage
• Anything-sync-daemon on Github

Monitoring internet connection stability/latency with Smokeping

You had definitely came across a situation when your internet connection was down for a while. It usually happens in the most critical moment when you're in the middle of something important. I was asking myself how often this kind of internet connection outage happens when I am not using the internet at the moment. I decided to create a long-term stats of the internet stability to have something as a solid proof when complaining to my ISP. And it's also interesting to know how reliable is my ISP compared to the others.

To find out the moment of the connection outage the link must be tested each few minutes. This is perfect job for headless Raspberry PI connected to your local network. By "testing the link" is meant that the latency and packet loss will be measured between your network and some remote server. As a remote server can be chosen any reliable service like Google, Facebook, Twitter or better some of your remote servers.

SmokePing

I found a perfect ready to use tool named SmokePing that comes pre-bundled with the Raspbian linux. It does exactly what we need – it monitors the network latency and plots all the results into charts. SmokePing consists of a background daemon process which organizes the latency measurements and a CGI script which presents charts via web browser.

SmokePing installation and setup

All commands in this tutorial will be run as root

sudo su

Install SmokePing with all its dependencies

apt-get install smokeping

After installation you can find all important config files in /etc/smokeping/config.d/ directory. Detailed documentation of the config options can be found at the SmokePing website.

At this point you may consider editing the general settings like your name, email, host in the /etc/smokeping/config.d/General, but this is not the point of this tutorial.

The most important thing is to configure probes – i. e. methods that will SmokePing use to measure the network latency. This tool supports a bunch of protocols to probe network connection (e. g. ICMP ECHO, DNS, HTTP, FTP, SSH). We will stick to the simplest one – ICMP ECHO (common ping command that measures packet round-trip time).

First we need to configure the probe method itself. That can be configured in /etc/smokeping/config.d/Probes file:

*** Probes ***

+ FPing
binary = /usr/bin/fping
step = 300 # repeat each 5 minutes
pings = 10 # send 10 ICMP packets

Next thing that needs to be configured is list of target remote servers. Edit the file /etc/smokeping/config.d/Targets. Example config could look like this:

*** Targets ***

probe = FPing

menu = Top
title = Raspberry Pi Internet Connection Monitor

+ Local
menu = LAN
title = LAN

++ router
title = 192.168.0.1
menu = router
host = 192.168.0.1

+ Internet
menu = Internet
title = Internet Access (Ping)

++ Google
title = Google
menu = Google
host = www.google.com

++ FB
title = Facebook
menu = Facebook
host = www.facebook.com

You can replace target hosts with any other of your choice. I also added one host from my local network (my router gateway) to be able to distinguish between local network failures and internet connection failure.

Now it's everything set up and ready for starting the SmokePing daemon with the updated configuration:

service smokeping restart

Web front-end

After a couple of minutes you should see first results in the web front-end. Open your browser and load http://[raspberry]/smokeping/smokeping.cgi. You should see similar website with latency charts.

Extending SD card lifetime

Smokeping uses RRDTool as internal database for measured data. And since the monitoring is continuous process it writes a new data to SD card every few minutes. That can contribute to SD card wear out pretty significantly. To avoid this issue I recommend to read another article about moving RRDTool data to volatile memory and extending your SD card lifetime Persistent RAM disk – extending SD card lifetime

Useful links

• SmokePing homepage
• http://blog.kugelfish.com/2013/05/raspberry-pi-internet-access-monitor.html

Installing LAMP with FastCGI, PHP-FPM and APC

If you are looking for a way how to utilize your Rasberry PI as a cheap web server, this article is written just for you. Although Raspberry's computation power isn't as good as your phone or even laptop, it is still enough to host a small website. To be more precise, small website with a low traffic.

In this blog post I would like to show you step by step tutorial how to set up web server with PHP and optionally MySQL on your Raspberry to gain the most possible performance.

This guide has been tested on Raspbian Wheezy operating system, but it should work with any other version of Raspbian as well.

Installing Apache with PHP

First you need to switch yourself to root:

sudo su

Install packages for Apache and PHP:

apt-get install apache2-mpm-worker php5-fpm

Installing FastCGI

Then you need to install FastCGI module for Apache, but because this one is not available as binary (.deb) package in Rasbian repository, it is needed to be compiled from the source.

Install all dependencies first:

apt-get install debhelper cdbs apache2-threaded-dev dpatch libtool libapr1-dev pkg-config

Add line to /etc/apt/sources.list to enable sourca packages repository:

deb-src http://mirrordirector.raspbian.org/raspbian/ wheezy main contrib non-free rpi

Then download libapache-mod-fastcgi source package, compile and then install from binary package:

apt-get update
apt-get -b source libapache-mod-fastcgi
dpkg -i libapache2-mod-fastcgi*.deb

Configuring Apache and PHP

Enable mod_actions:

a2enmod actions

Create directory /var/www/fastcgi and make sure it's accessible for user www-data

mkdir /var/www/fastcgi
chown www-data:www-data /var/www/fastcgi

Edit file /etc/apache2/mods-available/fastcgi.conf to look following way:

<IfModule mod_fastcgi.c>
  AddHandler fastcgi-script .fcgi
  #FastCgiWrapper /usr/lib/apache2/suexec
  FastCgiIpcDir /var/lib/apache2/fastcgi

  Alias /php5.fastcgi /var/www/fastcgi/php5.fastcgi
  AddHandler php-script .php
  FastCGIExternalServer /var/www/fastcgi/php5.fastcgi -socket /var/run/php5-fpm.sock
  Action php-script /php5.fastcgi virtual

  # This part is not necessary to get it to work, but it stops anything else from being
  # accessed from it by mistake or maliciously.
  <Directory "/var/www/fastcgi">
    Order allow,deny
    <Files "php5.fastcgi">
      Order deny,allow
    </Files>
  </Directory>
</IfModule>

Installing APC

Your server will work even without this module, but usage of APC can significantly speed up your PHP applications. I personally tried with default Wordpress installation and with APC the response time was about 200 % faster. The APC (Alternative PHP Cache) is a free and open opcode cache for PHP. Its goal is to provide a free, open, and robust framework for caching and optimizing PHP intermediate code. It just mean that the PHP code isn't parsed everytime is needed again and again, but instead it's saved in cache and parsed only first time. Which is of course huge performance improvement.

The only drawback is that APC is not very well compatible with common php-cgi setup, because the cache is not shared among all spawned PHP processes. That's reason I used mod_fastcgi and PHP-FPM instead.

Let's install APC:

apt-get install php-apc

Installing MySQL

When needed you can optionally install MySQL server. Installation is straightforward:

apt-get install mysql-server mysql-client php5-mysql

Finishing up

Restart Apache and PHP-FPM to reload new configuration:

service apache2 restart && service php5-fpm restart

Add your website source code to /var/www