What: BarCamp Shanghai 2009
When: Sunday, June 14, 11:00am – 05:00pm
Where: The NetCircle HQ
What: BarCamp Shanghai 2009
Alvaro Videla, one of our developers here at The NetCircle, developed FireSymfony and released it in September 2009. FireSymfony is a Firebug/Symfony extentions that allows developers to see their Symfony debug toolbar as part of the Firebug extension, which leaves the page layout unchanged and also brings more UI improvement.
Alvaro is frequently updating the plugin and is working on new features for the upcoming versions.
One of our developers, Alvaro Videla, released a few weeks ago a Firebug/Symfony extension: FireSymfony. This allows developers to see their Symfony debug toolbar as an Firebug extension, which leaves the page layout unchanged and also brings more UI improvement. This being quite useful for us at the office.
Our support to this extension is part of our company culture; we encourage people to participate and innovate for the company or the community. We will provide resources to Alvaro like design for the plugin and the blog, hosting for the site and time during business hours where he can work on documentation, bug fixing and new features.
Good luck little plugin!
Today I was checking on one of my favorite blog and scrolling over the comments I had a relevation… There is a patch (a simple text file) which adds data replication to Memcached. It is called Repcached and it is the best thing ever… since the frozen fridge!
I mean Memcache is a very important software when dealing with high loaded website and we use it a lot in our projects. Your script is overwhelmed by slow queries?! No problem, just put a Memcache in between, store the results once and enjoy them until the expiration date. It is all about speed as it is only stored in-memory; and then there is no safety, no persistency. But it is ok as long as it is used as query caching for example; if the data is not stored in Memcache, it can still be found in the database.
The problem is more when using Memcache to store data that can’t be easily regenerated, like very complex and slow query (which will kill your app just by running during business hours) or user session. The later was a big concern for us. There are others alternative than Memcache to store session data but…
- File storage is not a solution anymore with a cluster of servers.
- Cookie storage is relatively unsafe and not adapted for our average session data size.
- Database is the most common solution but performance quickly can become a big issue when dealing with so much update queries (table locking, transaction overhead…).
Memcache was then chosen and quickly prove its efficiency at frequently delivering and updating data. But there is no persistency and by growing too fast the service starting to behave strangely; kicking out some users of the site randomly… Not nice.
Then began a month of uncertainty, watching Ganglia reports every morning, trying to find a pattern in those mysterious logs out. By profiling connection data, tuning (slab size especially) and partitioning the sessions between 2 instances (from the client code), we then came back to a normal behavior. And despite restarting them every day (for memory cleaning), we didn’t have any problem with it since 1 year already. Still I remember those times when Memcache died…
But yeah replication is there for Memcache and it solves all those problems. Basically this patch will allow 2 instances to replicate the data between each other. And then if one instance dies, the data will still be available on the other instance. There are some “nice” drawings explaining the whole process on the Repcached wiki. Raw, simple and powerful.
As a bonus, it then makes perfectly sense in PHP to use the function Memcache::addServer. This creates a pool of connection, in which you can retrieve data and then will handle naturally the replication. Switching from one instance to another in case one is unavailable. The users are now safe.
Hi, I am Boby, a frenchie working for TheNetcircle since 3 years and i act now as technical lead. So this is my first article and i hope the beginning of a long serie of technical ones. I won’t be the only one writing in this category; i will try to push other workmates to share their knowledge and experience. For this article i am going to talk about our main issue at the moment, which solution we chose and why. Hope you will enjoy it.
We are currently busy developing a new version of our main website, a very special dating community. It is quite a success, we have 2 millions registered users and at peak hours around 20,000 concurrent users browsing the site. To serve this traffic we are using 50 machines like databases, static servers, php servers… All those have to be organized to achieve scalabilty and a kind of high availability.
Since 2 years we are using a very traditional setup. We wanted to separate the traffic between static and dynamic content, as it is better to optimize. We had then 2 clusters: one for static content using Lighttpd and one for dynamic content using Apache server. For each cluster there is a loadbalancer in front, currently LVS. And on top of that we have several Squids which forward the requests to the appropriate cluster. The Squid in this case was just using as reverse proxy and forwarding, no caching as the Lighttpds are fast enough to deliver the content (in fact putting a squid in front of them was more a drawback than anything else). Some static content is served directly from the loadbalancer using another domain name; still for AJAX requests the static content has to be under the same domain name.
Here is the layout as described:
Every layer scales; the Squids by DNS round robin, the rest by LVS and up to now this was working fine. Still it is a lot of physical servers and it would be nice to reduce it to ease the maintenance and reduce our expenses; especially with the new version of the site coming, we could spend some time experimenting.
We started by the Squid. By looking at his load we noticed that it was only using one CPU, letting the other one collecting dust. Squid is not multi-threaded and except launching 2 instances of it, there won’t be any benefit of running it on any modern CPUs. We then looked for other solutions; first very old redirection daemons, then incomplete Squid wannabes and finally lightweight HTTP servers. That is where we found NginX.
NginX is, as they said, “a free, open-source, high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server”. As we heard before by browsing some ruby forums, “Nginx is known for its stability, rich feature set, simple configuration, and low resource consumption”. The best of Russian hacking, that is so badass; we had to try it. After playing with it a while, we noticed that not only it will replace efficiently our forwarding Squids but its features could cover most of our requirements for the site.
- Request forwarding: That was our main focus at first and there is a module rewrite. It is using a simple but elaborate syntax which allow conditional statement, include of a set of rules and real POSIX REGEX rules. This was a big change compared to pseudo REGEX rules used by Asqredir as the Squid used to be configured. After rewriting our current rules, we had a more organized configuration and no rules duplication. For sure there was some limitations as the rewriting language is sometimes too simple (not more than 9 variables can be used), but nothing blocking or which would have killed our enthusiasm.
- Load balancing: Simple loadbalancing is natively included in NginX. For each domains you want to loadbalance you specified the IP and here you go. Bye bye LVS.
- Static server: As Lighttpd, NginX is an asynchronous lightweight HTTP server. So it will just deliver as fast as it can any static content and no need for caching. Adios Lighttpd.
Here is what the layout looks like then:
Much simpler and less machines, which means less maintenance and less hosting fees. Several NginX with DNS round robin, also IP fail over to be safe and it is ready for production. Well we still need more test, configuration tuning but from what we have seen we are confident.