In order to moderate pictures from Skyrock Blog, moderators have dedicated web servers, that send pre-generated pages with 500 thumbnails to be reviewed.
Serving these static pages is not something that requires a lot of horsepower. Even with Apache web server, little memory and a single IDE disk, the system load on these hosts had always been very low and everything worked as expected.
But yesterday, something went wrong. I got a call “images are damn slow to load, moderators can’t do their job”.
Indeed, the system was very slow. While about 15 Apache processes were usually running, there were 1000. The filesystem was slow. All processes were waiting for the disk. Very odd.
The reason: while most moderators were previously working with Opera and Firefox, a lot of them switched to Internet Explorer.
Internet Explorer was the server killer. In order to load the 500 pictures, Internet Explorer opened 500 connections to the same host, in order to load all pictures at the same time. Every moderator using IE loaded the server the same way. Every picture was different so the disk cache was pointless. The disk was totally unable to serve so much random data at the same time. Of course, keepalive was enabled on the server, but Internet Explorer didn’t care.
A workaround would be to tile the thumbnails into a single picture and to use offset and clipping to display a thumbnail.
But as I was in a hurry, I restricted the service to non-IE users. All moderators switched back to Firefox. Firefox is mature enough to restrict the number of concurrent connections to the same server. The server load immediately dropped to nothing, everything was smooth again, and everyone could do his job as before.
A workaround is to use a proxy like Squid, but it would be so much better if IE behaved a bit gentlier.