Server Outage

Saga Musix · February 17, 2011, 18:48:59

Some of you might have noticed a server outage this noon at around 12:45 UTC. The server was shut down by the provider because it was consuming all resources. As far as I can see, there was no security breach; a problem with the mail server is the most plausible explanation at the moment. At this time of writing I cannot guaranetee that the server was not cracked (but I'd be surprised if it was, since all software is kept up-to-date, the last time I checked for system updates was actually yesterday), but if you want to feel really safe, you can of course change your passwords on the forum / issue tracker / etc. - I don't think it's necessary at the moment, as passwords on the forum and issue tracker are salted anyway, but of course it's a good idea in general to change passwords reguarily. Right now I'm looking into the mail server configuration to see if a bad configuration was indeed the reason for this server outage.

Rakib · February 17, 2011, 22:00:01

Do you use the mail server?
If not how about closing the mail server permanently?

Saga Musix · February 17, 2011, 22:08:52

The mail server is needed to send out confirmation mails and similar stuff; anyway, there was a rather silly misconfiguration which has been fixed now, so if the mail server was part of the problem or even caused it, this should not happen anymore. I still wonder a bit about a traffic peak (about 14MiB of in/out traffic combined in one hour, which doesn't sound much but actually is, in comparison to the usual site traffic) and excessive usage of CPU and/or RAM which occurred around the same time (probably caused by the mail server fuckup), but seeing that a similar situation was caused by a very stupid spider earlier tonight (a fix for this has also been applied by means of installing a proper robots.txt for the wiki), it doesn't seem too alarming anymore.

LPChip · February 18, 2011, 07:12:13

You're doing a great job Jojo!

Thanks for taking care of this.

Saga Musix · February 18, 2011, 13:11:36

Also looking at the forum stats right now, I'm rather confident that the outage was caused by some aggressive bot hammering ("Most Online Ever: 193 (Yesterday at 13:46:45)") - that is a whole lot more than usual, and that would also explain the traffic peak as well as the use of resources. I might have to tweak some Apache settings...

Rakib · February 18, 2011, 13:18:24

Cheap trick is to set a maximum clients to visit the site at once, but I don't know how popular the forum actually is.

http://httpd.apache.org/docs/trunk/mod/mpm_common.html#maxclients

Saga Musix · February 18, 2011, 14:21:55

MaxClients is actually set to a good value, and the website uses caching and similar techniques for reducing load actually, so I'm not sure if it would help much to lower the value even more. If a similar congestion happens again, further actions will be necessary, but at the moment I'll just watch the situation and see if anything changes.