Install Memcached With repcached “Built-In Server Side Replication”

Everyone probably know about memcached (http://memcached.org/) and its high performance name-value based memory object cache interface. Its main purpose is to provide an easy to use distributed caching engine in a multinode environment. However, have you ever wanted to let memcached handle replication?

If you’d like to add high-availability capabilities, you’re advised to let the client (-library) handle replication of your data throughout all nodes. Let’s say using memcached as storage backend for php sessions is configured as follows:

# configuration for php memcache module
extension=memcache.so
session.save_handler = memcache
session.save_path = "tcp://10.0.2.10:11211?persistent=1,tcp://10.0.2.11:11211?persistent=1"

Now sessions get distributed (not replicated) through both nodes. As soon one node goes down you’ll loose all that data. As long you only use memcached as a performance booster and distributor of some “cacheable” data that should not hurt: The lost data gets filled up as they did whilst creating the cache entries.

To replicate your session data you’ll need to use a custom session handler that connects to each of these nodes on its own… for replication. Same if you’d like to write some own cached objects to your beloved memcached. Still your application needs to know all nodes to write to.

Settle down with replication by letting memcached work it out as repcached

These issues might confuse developers that just want to focus on their application and don’t want to care so much about HA, load-balancing and scalability. With repcached at least a 2-node HA-Solution can be setup easily and transparently.

A simple patch adds fancy master-master replication to your memcached. Now you’ll only need to connect to your local memcached which than handles replication on its own.

So you’re application may remain dump and simple, using your local POSIX filesystem (which might be replicated by DRBD, GlusterFS, etc) your local Database (you guessed it: “which handles replication on its own”) and connecting to a local instance of your “memcached-cluster”. That setup opens up some more simple and transparant options for an easy and lightweight HA-Scalability.

Installation, configuration and testing the cluster

First grab a copy of the repcached patch or even download the pre-patched memcached-source from http://repcached.lab.klab.org/. There is a dependency on libevent which might need to be resolved prior to installation.

root@memrep-01 ~ # yum install libevent-dev

Now unpack and cd into the directory:

root@memrep-01 ~ # tar xvf memcached-1.2.8-repcached-2.2.tar
root@memrep-01 ~ # cd memcached-1.2.8-repcached-2.2/

Configure with –prefix=/usr/local/memcached –enable-repclication options.

root@memrep-01 ~/memcached-1.2.8-repcached-2.2 # ./configure –prefix=/usr/local/memcached –enable-replication
root@memrep-01 ~/memcached-1.2.8-repcached-2.2 # make
root@memrep-01 ~/memcached-1.2.8-repcached-2.2 # make install

Don’t worry too much about a previously installed memcached package. This manual installation will just install memcached in /usr/local/memcached letting your “original” memcached untouched in /usr/bin/memcached.

In contrast to the already installed package you might like to provide command-line options by using a simple default configuration in /etc/sysconfig/memcachedrep (rep => ‘replication’), ie:

default configuration on Linux variant

PORT="11211"
USER="nobody"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS="-x 10.0.2.10"

With that in place an init script is set up quite easy by using and modifying the init script skeleton:

init-script for Linux

#! /bin/sh
#
# chkconfig: - 55 45
# description:  The memcached daemon is a network memory cache service.
# processname: memcached
# config: /etc/sysconfig/memcachedrep

# Source function library.
. /etc/rc.d/init.d/functions

PORT=11211
USER=nobody
MAXCONN=1024
CACHESIZE=64
OPTIONS=""

if [ -f /etc/sysconfig/memcachedrep ];then
. /etc/sysconfig/memcachedrep
fi

# Check that networking is up.
if [ "$NETWORKING" = "no" ]
then
exit 0
fi

RETVAL=0
prog="/usr/local/memcached/bin/memcached"

start () {
echo -n $"Starting $prog: "
# insure that /var/run/memcachedrep has proper permissions
chown $USER /var/run/memcachedrep
daemon $prog -d -p $PORT -u $USER  -m $CACHESIZE -c $MAXCONN -P /var/run/memcached/memcachedrep.pid $OPTIONS
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/memcachedrep
}
stop () {
echo -n $"Stopping $prog: "
killproc `cat /var/run/memcachedrep.pid`
RETVAL=$?
echo
if [ $RETVAL -eq 0 ] ; then
rm -f /var/lock/subsys/memcachedrep
rm -f /var/run/memcachedrep.pid
fi
}

restart () {
stop
start
}

# See how we were called
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status memcached
;;
restart|reload)
restart
;;
condrestart)
[ -f /var/lock/subsys/memcachedrep ] && restart || :
;;
*)
echo $"Usage: $0 {start|stop|status|restart|reload|condrestart}"
exit 1
esac

exit $?

Follow the above steps on second node. Make sure you make the appropriate changes for the IP in /etc/sysconfig/memcachedrep.

Solaris

On Solaris you need to add option -l <ip address> to make replication work and run it in daemon mode. You will need to create the following:

  1. Method Script [Location: /lib/svc/method/memcached]
  2. Manifest File [Location: /var/svc/manifest/network/memcached.xml]

Solaris svc method script

#!/bin/bash
. /lib/svc/share/smf_include.sh

MCBIN="/usr/local/memcached/bin"
MCOPTIONS="-m 64 -u www -l 192.168.5.133 -p 11211 -x 192.168.5.134"

if [ -x ${MCBIN}/memcached ]; then
${MCBIN}/memcached ${MCOPTIONS} &
else
echo "memcached is missing or not executable."
exit $SMF_EXIT_ERR_CONFIG
fi

exit $SMF_EXIT_OK

Once file is set the executable rights.

# chmod +x !$

Create the Manifest file.

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">

<service_bundle type='manifest' name='memcached' >
<service name='application/database/memcached' type='service' version='1' >
<create_default_instance enabled='false' />

<dependency
name='network'
grouping='require_all'
restart_on='none'
type='service'>
<service_fmri value='svc:/milestone/network:default' />
</dependency>

<dependency
name='name-services'
grouping='require_all'
restart_on='none'
type='service'>
<service_fmri value='svc:/milestone/name-services:default' />
</dependency>

<exec_method
type='method'
name='start'
exec='/lib/svc/method/memcached'
timeout_seconds='10' />

<exec_method
type='method'
name='stop'
exec=':kill'
timeout_seconds='60' />

<exec_method
type='method'
name='refresh'
exec=':kill -HUP'
timeout_seconds='60' />

<property_group name='memcached' type='application'>
<propval name='options' type='astring' value='' />
<propval name='value_authorization' type='astring' value='solaris.smf.value.memcached' />
<propval name='action_authorization' type='astring' value='solaris.smf.manage.memcached' />
</property_group>

<stability value='Evolving' />

<template>
<common_name>
<loctext xml:lang='C'>memcached 1.2.8 </loctext>
</common_name>
<documentation>
<manpage title='memcached' section='1m' manpath='/usr/share/man' />
</documentation>
</template>

</service>
</service_bundle>

Once the manifest file is created validate the file with following command:

# svccfg validate !$

If  everything seems OK (i.e. no error’s are displayed on console) import the manifest file with following command:

# svccfg import !$

These steps also need to be followed on second Solaris node. Make sure you make the appropriate changes for the IP in /etc/local/lib/svc/method/memcached.

Enabling the service in Solaris:

To enable the memcached service run following command as root.

# svcadm enable memcached

Once again on the prompt check the service has not failed or is in maintenance.

# svcs -xv

Testing

After setting up repcached on both nodes you should probably test that out. You might connect to memcached on each node per telnet:

root@memrep-01 ~ # telnet 127.0.0.1 11211

Trying 127.0.0.1…
Connected to 127.0.0.1.
Escape character is ‘^]’.

Now issue a set command to write some test values just on one node:

set var 0 0 3
val
STORED
get var
VALUE var 0 3
val
END

move over to the other node, connect and try to read:

root@memrep-02 ~ # telnet 127.0.0.1 11211

Trying 127.0.0.1…
Connected to 127.0.0.1.
Escape character is ‘^]’.
get var
VALUE var 0 3
val
END

Try that vice-versa and have fun! And finally tear down one of these nodes after you pushed some more values into memory. The running node still keeps responding with _all_ data – excellent.

The real fun part is, that next as soon you start up the unavailable node again it gains all the “lost” data from the other master. Go ahead try on your own. Same will happen if you now tear down the other node and bring it back up again. Even data that was set to one node whilst the other node was down will get added to the upcoming node after startup.

This setup is capable of a 2-node-replication setup only – yet.

Memcached Repcached Options explanation

-m To define amount of memory to be used by memcached.

-l To define listening IP address of memcached.

-p To define port on which memcached will be listening.

-u To define user memcached will run as. Memcached should run as normal user not as super user.

-x To define master replication server IP address.

2 Comments

  1. Hi, Thanks very much for your explanation. Would you let us know, how to replicate cache datain more than 2 memcache instances or any other hacking implementation ?
    Thanks

Leave a Reply

Your email address will not be published.


*


CommentLuv badge