Sunday, August 7, 2016

Persistent RAM disk – extending SD card lifetime

Internet forums are full of stories about SD card corruption and data loss when running Raspberry PI. It happened to me as well and obviously I wasn't very happy about that. It took me a while to recover all the data from backups and make the PI running again. I decided to find the way how to avoid this problem in the future or at least reduce the chances of card being corrupted.

The overall problem is called SD card wear out. The main point is flash memory media have erasable cells, each of which can be put through a limited number of erase cycles before becoming faulty. Usually it takes thousands of cycles which may look like a huge number, but it depends what are you doing with you Raspberry. If your app is not designed carefully or worse you're using persistent database, thousands of erase cycles can be depleted within few days.

I came across this problem in discussion about Monitoring internet connection with Smokeping. Smokeping uses RRDTool as internal database for measured data. And since the monitoring is continuous process it writes a new data to SD card every few minutes. That can contribute to SD card wear out pretty significantly.

It's crucial to reduce write operations to SD card. The obvious solution is to use RAM disk or tmpfs (i. e. file system that stores all the files in volatile memory instead of persistent storage). But problems will arise when you realize you'll lose all the data during reboot. For example if we move /var/log to tmpfs we'll always start with empty directory after reboot. That won't work very well since certain apps expect some specific directory structure within /var/log and they'll crash at startup.

More advanced solution is to combine tmpfs with persistent storage effectively creating persistent ramdisk. So the life-cycle of data will look like this:

  • After boot data from chosen directory is copied from SD card to tmpfs
  • When data is modified it happens at no cost in RAM
  • To prevent data loss during electricity outage or random restart there is hourly back-up made to SD card
  • When properly shutting down all actual data from tmpfs are copied back to SD card

Anything-sync-daemon

I found a great tool called Anything-sync-daemon written in bash that does exactly what we need – maps chosen directories to tmpfs and syncs changes back to SD card when needed.

Unfortunately Anything-sync-daemon is available only for Arch linux so I decided to compile and build my own package for Debian/Raspbian 8 Jessie.

Anything-sync-daemon package for Debian

All commands need to be run as root, thus sudo su

Install required dependencies for building a package and rsync

apt-get install git build-essential checkinstall rsync 

Clone the current source code of Anything-sync-daemon from Github

git clone https://github.com/graysky2/anything-sync-daemon.git

Open the source code directory and checkout the current version

cd anything-sync-daemon/ && git checkout v5.81

Build the Debian package and install it

make && checkinstall --pkgversion 5.81 make install-systemd-all

Now it's time to setup directories that need to be managed by asd. Edit /etc/asd.conf

WHATTOSYNC=('/var/lib/smokeping' '/var/log')
VOLATILE="/dev/shm"
  • WHATTOSYNC – List of directories that are mapped to RAM
  • VOLATILE – Path of the currently mounted volatile memory (tmpfs)
And finally restart the Anything-sync-daemon and enable it to start after boot.
systemctl enable asd.service && service asd restart

Debugging Anything-sync-daemon

There is an option to print the current configuration and state of monitored directories
# asd p

Anything-sync-daemon v5.81 on Raspbian GNU/Linux 8 (jessie)

 Systemd service is currently active.
 Systemd resync service is currently active.
 Overlayfs technology is currently inactive.

Asd will manage the following per /run/asd.conf settings:

 owner/group id:     smokeping/115
 target to manage:   /var/lib/smokeping
 sync target:        /var/lib/.smokeping-backup_asd
 tmpfs target:       /dev/shm/asd-smokeping/var/lib/smokeping
 dir size:           12M
 recovery dirs:      none

Useful links

7 comments:

  1. Have you considered writing data to a USB thumb drive or HDD as an alternative to the SD card? I would think this would reduce the SD card write cycle wear out.

    Great blog, just starting to explore Pi; your articles are very practical and simple.

    ReplyDelete
    Replies
    1. Thank you for your comment. I appreciate the interest. I agree that using HDD via USB would solve the issue, but it makes more noise and the Pi is not that compact with connected USB accessories.

      Also I am not sure about using USB flash drive. I believe SD cards and flash drives use the same type of flash memory, so the data corruption would appear sooner or later anyway.

      Delete
  2. Hello,

    thx for the great info shared overhere.
    Based on your related post, I did install Smokeping on my PiZero as I run into serious problems with my Internet connexion... but failled to install Anything-sync-daemon!
    When git clone https..., I'm being asked for Username & password!
    I'm quite new in this environment... I would appreciate some good advice ;-)

    Thx & rgds

    ReplyDelete
    Replies
    1. I don't really know. When I try to clone the Github repository it works without credentials. Anyway you can try to download release version in an archive:

      wget https://github.com/graysky2/anything-sync-daemon/archive/v5.81.zip && unzip v5.81.zip

      Then continue with building the package.

      Delete
  3. Found this article about tmpfs and pis http://www.zdnet.com/article/raspberry-pi-extending-the-life-of-the-sd-card/

    Was just wondering what you fstab looked like for you setup

    tmpfs /tmp tmpfs defaults,noatime,nosuid,size=100m 0 0
    tmpfs /var/tmp tmpfs defaults,noatime,nosuid,size=30m 0 0
    tmpfs /var/log tmpfs defaults,noatime,nosuid,mode=0755,size=100m 0 0
    tmpfs /var/run tmpfs defaults,noatime,nosuid,mode=0755,size=2m 0 0
    tmpfs /var/spool/mqueue tmpfs defaults,noatime,nosuid,mode=0700,gid=12,size=30m 0 0

    Cheers

    ReplyDelete
    Replies
    1. There is nothing added in /etc/fstab related to this setup. Anything-sync-daemon redirects the chosen directories after startup to /dev/shm.

      For example the mentioned /var/lib/smokeping is linked to /dev/shm/asd-smokeping/var/lib/smokeping

      $ ls -la /var/lib/smokeping
      /var/lib/smokeping -> /dev/shm/asd-smokeping/var/lib/smokeping

      Delete
  4. I get an error when tryoing to install checkinstall on a Raspbery Pi 4:

    apt-get install checkinstall
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    E: Unable to locate package checkinstall

    Do I have to add a repository to install it?

    ReplyDelete