Our users are safe! Memcache is replicated!

by Boby

Today I was checking on one of my favorite blog and scrolling over the comments I had a relevation… There is a patch (a simple text file) which adds data replication to Memcached. It is called Repcached and it is the best thing ever… since the frozen fridge!

I mean Memcache is a very important software when dealing with high loaded website and we use it a lot in our projects. Your script is overwhelmed by slow queries?! No problem, just put a Memcache in between, store the results once and enjoy them until the expiration date. It is all about speed as it is only stored in-memory; and then there is no safety, no persistency. But it is ok as long as it is used as query caching for example; if the data is not stored in Memcache, it can still be found in the database.

The problem is more when using Memcache to store data that can’t be easily regenerated, like very complex and slow query (which will kill your app just by running during business hours) or user session. The later was a big concern for us. There are others alternative than Memcache to store session data but…

  • File storage is not a solution anymore with a cluster of servers.
  • Cookie storage is relatively unsafe and not adapted for our average session data size.
  • Database is the most common solution but performance quickly can become a big issue when dealing with so much update queries (table locking, transaction overhead…).

Memcache was then chosen and quickly prove its efficiency at frequently delivering and updating data. But there is no persistency and by growing too fast the service starting to behave strangely; kicking out some users of the site randomly… Not nice.

Then began a month of uncertainty, watching Ganglia reports every morning, trying to find a pattern in those mysterious logs out. By profiling connection data, tuning (slab size especially) and partitioning the sessions between 2 instances (from the client code), we then came back to a normal behavior. And despite restarting them every day (for memory cleaning), we didn’t have any problem with it since 1 year already. Still I remember those times when Memcache died…

But yeah replication is there for Memcache and it solves all those problems. Basically this patch will allow 2 instances to replicate the data between each other. And then if one instance dies, the data will still be available on the other instance. There are some “nice” drawings explaining the whole process on the Repcached wiki. Raw, simple and powerful.

As a bonus, it then makes perfectly sense in PHP to use the function Memcache::addServer. This creates a pool of connection, in which you can retrieve data and then will handle naturally the replication. Switching from one instance to another in case one is unavailable. The users are now safe.

Posted in Development and tagged . Bookmark the permalink.


Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Join Our Team in Shanghai

Now hiring PHP Developer in Shanghai